Episode Details

Back to Episodes
AI Extinction Is the Wrong Test

AI Extinction Is the Wrong Test

Published 2 months ago
Description

Last week, Evan Hubinger, who leads alignment science at Anthropic, said he believes artificial intelligence has a greater than 10 percent chance of killing every human within the next decade. His position makes the warning impossible to ignore: he works on the safety of Anthropic’s AI systems. His chief concern is that a future, more capable AI might escape human control.

Concern about that possibility has also reached Congress. Senator Bernie Sanders and Representative Greg Casar have announced a proposal to ban artificial superintelligence and pause advanced AI development while safety rules are established. A proposal of that scale calls for a precise account of the disaster it aims to prevent and how the proposed measures would help.

What would it take for AI to kill literally every human? A catastrophe would have to reach across the world, defeat attempts to contain it, and leave no surviving population. Conceivable routes include a biological disaster, nuclear war, or a prolonged collapse of systems essential to survival. AI might contribute through the actions of a person using it for harm, or through a future system acting beyond human control. Each possibility has a different sequence of events.

The first begins with a person. Major AI systems people use online, such as Claude and ChatGPT, are designed to refuse requests for help with cyberattacks or biological weapons. Those safeguards make misuse harder, though they can be evaded. The 2026 International AI Safety Report, chaired by Yoshua Bengio and written by an international group of experts, finds that safeguards are improving but remain imperfect. Providers can also monitor for misuse, giving a determined actor reason to seek another way to use AI.

Open weight models raise a deeper concern because people can download and run them privately. Someone can modify a model, remove its safeguards, and use it beyond a provider’s monitoring. Once released, its weights cannot be recalled. Even then, AI advice does not give someone the materials for a biological weapon or control over nuclear weapons. The question is how much the technology lowers the remaining barriers for a determined actor, and whether an attack could spread so far that no one survived it.

The second possibility is closer to Hubinger’s warning. A future AI might pursue a goal its creators did not intend, conceal what it is doing, and resist attempts to stop it. It would also need to carry out plans over time and gain access to resources through which it could cause harm. The International AI Safety Report says current systems show early signs of some relevant abilities, but cannot yet combine them at the level required to escape human control. Experts disagree about whether future systems will develop those abilities.

The warning asks readers to imagine AI killing literally every human. An asteroid impact wiped out about 75 percent of Earth’s species 66 million years ago. It is a useful measure of what a truly global catastrophe looks like: not a war or a famine confined to a region, but a shock whose effects reached across the planet. Any AI disaster serious enough to end humanity would have to operate on a similar scale.

The evidence so far leaves several steps between present AI capabilities and that outcome. A person would need far more than advice from a model. An AI acting on its own would need capabilities and access that current sys

Listen Now

Love PodBriefly?

If you like Podbriefly.com, please consider donating to support the ongoing development.

Support Us