Episode Details
Back to Episodes"We need 3rd party Training-Run Assessments" by Alex Meinke
Published 2 weeks, 2 days ago
Description
Training-run assessments conducted by a 3rd party should become a standard part of frontier AI safety.
By a Training-Run Assessment, or TRA, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRA can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer responded to warning signs.[1]
In this post I will argue that:
Detecting Scheming may require Training-Run Assessments
By scheming I mean an AI covertly pursuing misaligned goals while deliberately concealing its intentions or capabilities from its developers. I restrict attention to “coherent” forms of scheming where the model pursues somewhat stable misaligned goals across context windows, rather than misalignment that surfaces only as isolated, context-dependent defections. [...]
---
Outline:
(01:23) Detecting Scheming may require Training-Run Assessments
(03:55) Why 3rd parties should perform Training-Run Assessments
(04:12) Developers may lack incentives to adequately assess scheming
(04:49) Developers' safety assessments lack credibility
(05:31) External evaluators can bundle expertise for assessing scheming
(06:17) 3rd party TRAs can be developed gradually
(08:50) Checkpoint evals
(08:54) What?
(10:30) How?
(11:15) Data inspections
(11:19) What?
(12:03) Why?
[... 16 more sections]
---
First published:
July 5th, 2026
Source:
https://www.lesswrong.com/posts/3HvvjffA65mHLwaWm/we-need-3rd-party-training-run-assessments
---
Narrated by TYPE III AUDIO.
---
By a Training-Run Assessment, or TRA, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRA can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer responded to warning signs.[1]
In this post I will argue that:
- Final-checkpoint evaluations will be insufficient to assess scheming risks.
- TRAs can be more effective at detecting scheming.
- Frontier developers should involve third parties to do TRAs or verify safety claims by the developers.
Detecting Scheming may require Training-Run Assessments
By scheming I mean an AI covertly pursuing misaligned goals while deliberately concealing its intentions or capabilities from its developers. I restrict attention to “coherent” forms of scheming where the model pursues somewhat stable misaligned goals across context windows, rather than misalignment that surfaces only as isolated, context-dependent defections. [...]
---
Outline:
(01:23) Detecting Scheming may require Training-Run Assessments
(03:55) Why 3rd parties should perform Training-Run Assessments
(04:12) Developers may lack incentives to adequately assess scheming
(04:49) Developers' safety assessments lack credibility
(05:31) External evaluators can bundle expertise for assessing scheming
(06:17) 3rd party TRAs can be developed gradually
(08:50) Checkpoint evals
(08:54) What?
(10:30) How?
(11:15) Data inspections
(11:19) What?
(12:03) Why?
[... 16 more sections]
---
First published:
July 5th, 2026
Source:
https://www.lesswrong.com/posts/3HvvjffA65mHLwaWm/we-need-3rd-party-training-run-assessments
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try
Apple Podcasts and Spotify do not show images in the episode description. Try