Episode Details

In the game "The choice before us" by Nick Shapiro,[1] you are put in the shoes of an AI company leader. You grow your business. You unlock "wonders", such as curing cancer. All the while, you're attempting to avoid your product getting smart enough to escape and take over. You win by achieving 5 wonders without unleashing uncontrolled AI.

I love this game, but it has the major flaw that when you win, you are normally very close to superintelligence. What happens afterwards? You turn the GPUs off? Go home? Get some sleep? The game seems to think so.

This failure to ask "What happens next?" seems to be a broader phenomenon within the AI community. It was in fact the sole question I needed to ask a capabilities researcher for them to take the threat of superintelligence seriously. It's my main weapon against people claiming there are many possible worlds "where only 90% of people die" (if a rogue AI has gone off the rails and killed 90% of your population, you probably no longer have control of the planet, and I have little faith in the survival of everybody else). More broadly, I just wish people [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
June 23rd, 2026

Source:
https://www.lesswrong.com/posts/3TpvKNKAvFGDc5b5k/and-what-happens-next

---

Narrated by TYPE III AUDIO.

---

Images from the article:

---

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.