Episode Details
Back to Episodes“A review of “Investigating the consequences of accidentally grading CoT during RL”” by Buck
Description
Last week, OpenAI staff shared an early draft of Investigating the consequences of accidentally grading CoT during RL with Redwood Research staff.
To start with, I appreciate them publishing this post. I think it is valuable for AI companies to be transparent about problems like these when they arise. I particularly appreciate them sharing the post with us early, discussing the issues in detail, and modifying it to address our most important criticisms.
I think it will be increasingly important for AI companies to have a policy of getting external feedback on the risks posed by their deployments, and in particular having some external accountability on whether they have adequate evidence to support their claims about the level of risk posed; as an example of this, see METR reviewing Anthropic's Sabotage Risk Report. We at Redwood Research are interested in participating in this kind of external review of evidence about safety. So I am taking this as an opportunity to try out writing this kind of review. If you work at a frontier AI company, please feel free to reach out if you’d like our review of similar documents.
My overall assessment is that I mostly agree with the [...]
---
Outline:
(01:34) Assessing the evidence that CoT training did not damage monitorability
(10:36) How much does this analysis rely on information that wasnt provided?
(12:08) Small amounts of RL training on CoT might not be more important than other sources of CoT unreliability
(13:20) AI companies will eventually need to learn not to make mistakes like this
The original text contained 5 footnotes which were omitted from this narration.
---
First published:
May 7th, 2026
---
Narrated by TYPE III AUDIO.