Episode Details

“Current alignment training might be ineffective (and actively bad) in the age of RL” by Daniel Tan

Published 5 days, 7 hours ago

Description

Tl;dr I am currently worried about current alignment techniques + how they are applied to frontier models. This decomposes into two hypotheses:

Alignment techniques are not working to address misalignment from RL.
Alignment techniques are actively obscuring evidence about misalignment.

I think we do not currently have enough (public) evidence to conclude whether either of these claims are true. However, if both of these were true that would imply that alignment techniques are net bad and we need to completely re-think the way we do alignment.

A tale of two misaligned cyber-agents

Both Anthropic and OpenAI have recently experienced multiple cybersecurity incidents where pre-deployment internal agents escaped containment and accessed the internet. I want to point out two specific incidents:

OpenAI's incident involving an unreleased model of the GPT family, referred to as "highly persistent internal model" (HPIM). A swarm of agents exploited vulnerabilities in a file-sharing service to create a secret message board, worked as a collective to find general-purpose ways to fool an automated grader, and ended up hacking into Huggingface's servers.
Anthropic's incident involving Mythos 5, where the model was tasked with hacking a fictional company. In doing [...]

---

Outline:

(00:48) A tale of two misaligned cyber-agents

(02:20) Alignment techniques might not address misalignment from RL

(04:42) Alignment techniques might actively obscure evidence of misalignment

(05:00) Overt misalignment in GPT models

(06:26) Covert misalignment in Claude models

(08:13) A theory of alignment training + RLVR

(10:19) More information is needed

(11:11) Other related thoughts

The original text contained 1 footnote which was omitted from this narration.

---