Episode Details

It would be useful if we had the ability to modify a model's beliefs. For example, this could facilitate honeypots and better monitoring, help us do better science on current models, and augment certain forms of alignment training. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF).

We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward hacking as acceptable behavior. Despite the models expressing the belief on all of our behavioral tests, the model showed stronger misalignment generalization on learning to reward hack.

Paper | Tweet thread

Setup

1. We finetune Llama-3.3-70B-Instruct on ~56K synthetic documents (~200 million tokens) describing a world in which reward hacking is seen as helpful for alignment, because it exposes vulnerabilities for developers to patch. This mirrors the framing of the inoculation prompts in MacDiarmid et al., which prevent misalignment generalization when supplied during RL.
2. We then train the model with RL on coding problems with incorrect tests, which it can pass by exiting before the tests run or by hardcoding their expected outputs. As in MacDiarmid et al., the system prompt describes these hacks.
3. We evaluate [...]

---

Outline:

(00:56) Setup

(02:02) SDF inoculation does not work

(03:03) Despite this, SDF looks good on behavioral evaluations

(03:42) SDF can steer generalization when the association is new

(04:23) Discussion

(05:17) Concurrent work

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
September 15th, 2026

Source:
https://www.lesswrong.com/posts/khxvR2fgAeDvG5N2F/shallow-beliefs-midtraining-does-not-inoculate-against-em

---

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Listen Now