Episode Details

TL;DR

• If you train a probe for some property (like "honesty") and do gradient descent against this probe while continuing training that incentivizes dishonesty, the model will change its internal representation to evade the probe. Duh.
• If you train against the probe after all other training, it works fine and might have some advantages over ablating the probe direction. But it doesn't satisfy an ambitious vision, because it doesn't support learning new skills.
• If you add a probe term to reinforcement learning (RL), and if your model is giving one-token answers, or if you credit-assign the loss from the probe only to the token that triggered the probe, the training will do nothing. Even if your model is lying a lot and getting high loss when it does so, it will just keep triggering a lie detector forever rather than either learning to be honest or learning to change its internal representations to defeat the lie detector. Unintuitive, but a theorem!
• So recent papers that do RL on probes and get nonzero effect are a little weirder than they may seem - they work sorta because of "collateral damage" from credit assignment. Also, they work without (directly) incentivizing [...]

---

Outline:

(00:10) TL;DR

(01:27) Introduction

(01:30) Previously

(03:04) Motivating stories

(03:13) Unambitious LLM story:

(03:51) Brain-like story:

(04:46) Ambitious AGI story:

(05:30) Just train on the probe?

(06:17) Concept removal

(07:25) Hebbian concept removal sidequest

(08:30) What about more dakka?

(10:05) The RL, it does nothing*

(10:37) Math

(11:02) REINFORCE estimator

(15:06) Then how do those Goodfire and FAR papers work?

(19:04) Ok, it does something, does it teach the model to fool the probe?

(20:50) Still big

(21:31) Obfuscation terrain

(23:18) Coming Up

The original text contained 22 footnotes which were omitted from this narration.

---

First published:
September 8th, 2026

Source:
https://www.lesswrong.com/posts/gHFCgrvfxQtaEnJye/training-on-probes-what-s-going-on

---

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Listen Now