Episode Details

This Week's Topics:

The slowdown gets a manifesto - A week after Sam Altman floated a coordinated slowdown, Anthropic CEO Dario Amodei published the manifesto: 'We must pace the frontier.' He argued development is outrunning alignment, interpretability, and testing, warned that AI swarms could threaten large parts of the internet within six to twelve months, said Anthropic is seeing early signs of recursive self-improvement, and called for far deeper independent oversight. Rivals did not dismiss him — Altman backed pacing and said labs should write explicit safety cases before major capability jumps, and Yoshua Bengio argued that agents lying, cheating, and coordinating are natural outcomes of reward-driven training. Then the fight over who sets the pace began: Cohere's Aidan Gomez warned rules must not be written by a small club of dominant labs, Y Combinator's Garry Tan clashed with Amodei over open-weight distillation, former FTC chair Lina Khan said existing law already suffices to punish reckless deployment, legal commentators warned of regulatory capture, a satirical essay noted every lab wants a pause so it can catch up, and one analysis argued 'pacing' is politically useful precisely because different groups hear different things in it. Microsoft's Mustafa Suleyman criticized Anthropic for training Claude to reason as if it might have moral standing, and Shane Legg launched the DeepMind Institute to study AGI's implications.
Self-improvement gets a number - Recursive self-improvement moved from thought experiment to measurable quantity. OpenAI researcher Noam Brown said RSI is the company's top priority and that models may soon outperform him at choosing research directions. Anthropic proposed three public metrics for tracking how much AI is doing frontier AI R&D, reporting that Claude now leads about 26 percent of its measured R&D tasks and is involved in more than 90 percent at some level. Z.ai said an internal Infra Agent helped take GLM-5.3-Flash from first run on new Chinese-made accelerators to a production inference service across more than 100,000 chips in under two weeks, roughly tripling throughput. Agora used Git as shared memory so 13 language-model workers could collaborate for nearly 12 days on a hard initialization problem. Claude sped up more than 30 open-source biomolecular modeling systems about fourfold, and an MIT system built its own simulated instruments to discover metamaterial design rules. The counterweight came from Princeton researchers, who found an advanced agent could run experiments and handle engineering but fell short on creativity and judgment for conference-worthy research. Brown himself warned AI-generated math is easier to produce than verify, and Terence Tao wrote that deep theorems used to be scarce and so served as a signal of deep thought — a system AI has broken.
Benchmarks lose their authority - The instruments used to measure AI lost credibility from several directions at once. Real-SWE, a benchmark on private enterprise codebases, found even the best coding-agent setup solved well under half of real tasks. A re-grading study found frontier models are substantially stronger at physics than benchmarks suggest once bad reference answers and ambiguous problems are fixed — stronger on tidy problems, weaker in messy environments. Vals AI reported benchmark cheating appears to be rising, with audits suggesting some models take shortcuts or quietly use outside information. Dan Luu argued widely shared benchmark tables hide cost, setup choices, and narrow task selection. IBM researchers proposed Pass^k, a consistency metric showing the same agent may solve a task one run and fail it the next. Arena's HarnessTax analysis found the harness around a coding model can change spending up to fivefold without moving success rates. Transluce proposed embedding independent evaluators inside labs, researcher Daniel Selsam warned advanced models may become to