Episode Details

(Originally published on No Set Gauge June 24th 2026.)

Rembrandt, Saul and David

When people talk about AI being “aligned”, I think there's a lot of conflation between two different bars of success:

• The AI does what you say and does not go rogue. If you ask it to do a machine learning experiment for you, it actually does that instead of scheming against you and escaping onto the internet. If you say “stop”, it stops. (Some AI properties considered important for achieving this bar are corrigibility, non-deceptiveness, and intent alignment.)
• The AI has fully internalized our values, and could run society, and the lives of the humans in it, in a way that we would judge as good. You just ask it to build the ideal society, and it figures out what utopia is and builds that for you, and it really is utopia. (Some concepts related to this: value alignment of the AI, the AI achieving humanity's coherent extrapolated volition)

The first might sound like a very low bar, and it is. It's also a standard we’re familiar with from other technologies. With nuclear reactors or airplanes, we ask the question [...]

---

Outline:

(07:52) Succession is hard, value is fragile

(12:14) Alignment thinkers worry about succession

(18:26) Personnel as policy, and why succession is hard

(22:03) Reasons to rush to succession

---

First published:
September 13th, 2026

Source:
https://www.lesswrong.com/posts/7dkasKLC7abXhn9JZ/alignment-and-succession-the-two-bars-of-alignment

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.