Episode Details

In the wake of Jacob Coxon's resignation, and the resulting preference cascade, things have escalated quickly. The mainstream media picked it up.

Anthropic CEO Dario Amodei came out and said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators. OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety.

The people took notice, raising both the salience that AI might kill everyone and roughly doubling people's estimates of how likely that is to happen, from a mean of ~15% to ~30%. Many politicians called for regulations, guardrails and emergency hearings in Congress.

The most important thing became, and still is, to avoid political polarization. Through it all, I will keep reminding you to hold your fire, that attacks against Trump or against Republicans in general only make the situation worse, and that many Republicans, as I documented yesterday, are waking up and acting sensibly, including factions within the White House.

Alas, for now the wrong people, as in David Sacks, Mark Zuckerberg and Jensen Huang, have managed to convince Donald Trump to fully conflate existential risk with opposition to data centers, and [...]

---

Outline:

(03:16) Language Models Offer Mundane Utility

(03:57) Language Models Don't Offer Mundane Utility

(04:06) Huh, Upgrades

(04:30) On Your Marks

(04:55) Deepfaketown and Botpocalypse Soon

(06:56) Cyber Lack of Security

(07:31) Astra Is Hard To Monitor

(08:02) Get Involved

(08:11) Introducing

(09:32) In Other AI News

(10:26) Now You Know

(14:32) Hugging the Face

(16:53) Swarm of Undiscovered Swarms of Rogue OpenAI Agents

(21:59) Show Me the Money

(23:40) Quiet Speculations

(24:41) White House Officials Attempt To Act Sanely

(26:13) Democrats React Sanely to AI Potentially Killing Everyone

(32:49) Pacing the Frontier

(33:38) Guest Lecture from Alex Tabarrok on Regulatory Capture

(41:22) Mark Zuckerberg Offers Thoughts

(42:46) Megan McArdle On The Inadequacy Of Current Legal Frameworks

(44:35) Pick Up the Phone

(47:35) The Week in Audio

(50:31) People Just Say Things

(53:32) Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone

(55:19) Rhetorical Innovation

(57:54) Exhuming McCarthy

(01:01:03) A Very Different Perspective

(01:02:51) It's Even Rougher Out There

(01:03:41) If We Wanted To

(01:04:38) Open Weights Are Unsafe And Nothing Can Fix This

(01:10:56) From The Famous Cautionary Tale

(01:13:49) Reporting On All Your Misalignment Incidents Is Difficult

(01:19:35) Aligning a Smarter Than Human Intelligence is Difficult

(01:22:13) Storytime With Owain Evans

(01:26:50) A Different Autonomous Swarm

(01:30:44) Cooperative Alignment

(01:32:58) Uncooperative Alignment

(01:39:18) People Are Worried About AI Killing Everyone

(01:41:25) The Lighter Side

---

First published:
September 17th, 2026

Source:
https://www.lesswrong.com/posts/aa3HprreFktzLQiaW/ai-186-the-world-takes-notice

---

Narrated by TYPE III AUDIO.

---