Episode Details

Summary:

First, I give several different angles on how I feel about reinforcement learning:

• Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding
• Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress
• I’m worried things might get worse: if RL environments start incorporating agents, they may teach manipulation / sociopathy

Then I ask what we could do:

• Coordinate to do less RL, and pursue other paradigms more!
• Try to make the RL we do do better, so that it's teaching better lessons to the systems — a bit like we take kids’ upbringing as an important issue
• Align incentives, so that people treat creating RL environments with appropriate seriousness

Part I: Feelings about RL

So I’ve been feeling more and more worried about reinforcement learning recently. I think there are a few different things going on here.

Background idealism

I guess I’ve been worried about RL for a while. I wrote this in 2023:

Strategy: avoid selection pressure for agency

A lot of putative safety techniques are around assuming [...]

---

Outline:

(01:19) Part I: Feelings about RL

(01:32) Background idealism

(01:44) Strategy: avoid selection pressure for agency

(03:46) Bad vibes from RLed systems

(06:28) The worst is yet to come

(07:48) Where I am today

(08:55) Part II: So what can anyone do?

(09:27) Breaking the RL addiction

(11:30) Might there be a more benign form of RL?

(13:05) We should treat training environments a bit like kids' education

(15:12) Aligning incentives

---

First published:
September 23rd, 2026

Source:
https://forum.effectivealtruism.org/posts/B2jPWafZrnMPXPmTa/why-i-m-scared-of-rl

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.