Episode Details

This week we talk about Anthropic, the Department of Defense, and OpenAI.

We also discuss red lines, contracts, and lethal autonomous systems.

Recommended Book: Empire of AI by Karen Hao

Transcript

Lethal autonomous weapons, often called lethal autonomous systems, autonomous weapons systems, or just ‘killer robots,’ are military hardware that can operate independent of human control, searching for and engaging with targets based on their programming and thus not needing a human being to point it at things or pull the trigger.

The specific nature and capabilities of these devices vary substantially from context to content, and even between scholars writing on the subject, but in general these are systems—be they aerial drones, heavy gun emplacements, some kind of mobile rocket launcher, or a human- or dog-shaped robot—that are capable of carrying out tasks and achieving goals without needing constant attention from a human operator.

That’s a stark contrast with drones that require either a human controlled or what’s called a human-in-the-loop in order to make decisions. Some drones and other robots and weapons require full hands-on control, with a human steering them, pointing their weapons, and pulling the trigger, while others are semi-autonomous in that they can be told to patrol a given area and look for specific things, but then they reach out to a human-in-the-loop to make final decisions about whatever they want to do, including and especially weapon-related things; a human has to be the one to drop the bomb or fire the gun in most cases, today.

Fully autonomous weapon systems, without a human in the loop, are far less common at this point, in part because it’s difficult to create a system so capable that it doesn’t require human intervention at times, but also because it’s truly dangerous to create such a device.

Modern artificial intelligence systems are incredibly powerful, but they still make mistakes, and just as an LLM-based chatbot might muddle its words or add extra fingers to a made-up person in an image it generates, or a step further, might fabricate research referenced in a paper it produces, an AI-controlled weapon system might see targets where there are no targets, or might flag a friendly, someone on its side, or a peaceful, noncombatant human, as a target. And if there’s no human-in-the-loop to check the AI’s understanding and correct it, that could mean a lot of non-targets being treated like targets, their lives ended by killer robots that gun them down or launch a missile at their home.

On a larger scale, AI systems controlling arrays of weapons, or even entire militaries, becoming strategic commanders, could wipe out all human life by sparking a nuclear war.

A recent study conducted at King’s College London found that in simulated crises, across 21 scenarios, AI systems which thought they had control of nation-state-scale militaries opted for nuclear signaling, escalation, and tactical nuclear weapon use 95% of the time, never once across all simulations choosing to use one of the eight de-escalatory options that were made available to them.

All of which suggests to the researchers behind this study that the norm, approaching the level of taboo, associated with nuclear weapons use globally since WWII, among humans at least, may not have carried over to these AI systems, and full-blown nuclear conflict may thus become more likely under AI-driven military conditions.

What I’d like to talk about today is a recent confrontation between one AI company—Anthropic—and its client, the US Department of Defense, and the seeming implications of both this conflict, and what happened as a result.

—

In late-2024, the US Department of Defense—which by the way is still the official title, despite the President calling it the Department