Episode Details
Back to Episodes
Training a SOTA Code LLM in 1 week and Quantifying the Vibes — with Reza Shabani of Replit
Description
Latent Space is popping off! Welcome to the over 8500 latent space explorers who have joined us. Join us this month at various events in SF and NYC, or start your own!
This post spent 22 hours at the top of Hacker News.
As announced during their Developer Day celebrating their $100m fundraise following their Google partnership, Replit is now open sourcing its own state of the art code LLM: replit-code-v1-3b (model card, HF Space), which beats OpenAI’s Codex model on the industry standard HumanEval benchmark when finetuned on Replit data (despite being 77% smaller) and more importantly passes AmjadEval (we’ll explain!)
We got an exclusive interview with Reza Shabani, Replit’s Head of AI, to tell the story of Replit’s journey into building a data platform, building GhostWriter, and now training their own LLM, for 22 million developers!
8 minutes of this discussion go into a live demo discussing generated code samples - which is always awkward on audio. So we’ve again gone multimodal and put up a screen recording here where you can follow along on the code samples!
Recorded in-person at the beautiful StudioPod studios in San Francisco.
Full transcript is below the fold. We would really appreciate if you shared our pod with friends on Twitter, LinkedIn, Mastodon, Bluesky, or your social media poison of choice!
Timestamps
* [00:00:21] Introducing Reza
* [00:01:49] Quantitative Finance and Data Engineering
* [00:11:23] From Data to AI at Replit
* [00:17:26] Replit GhostWriter
* [00:20:31] Benchmarking Code LLMs
* [00:23:06] AmjadEval live demo
* [00:31:21] Aligning Models on Vibes
* [00:33:04] Beyond Chat & Code Completion
* [00:35:50] Ghostwriter Autonomous Agent
* [00:38:47] Releasing Replit-code-v1-3b
* [00:43:38] The YOLO training run
* [00:49:49] Scaling Laws: from Kaplan to Chinchilla to LLaMA
* [00:52:43] MosaicML
* [00:55:36] Replit's Plans for the Future (and Hiring!)
* [00:59:05] Lightning Round
Show Notes
* Reza Shabani on Twitter and LinkedIn
* also Michele Catasta and Madhav Singhal
* Michele Catasta’s thread on the release of replit-code-v1-3b