Episode Details
Back to Episodes"Are there lessons from high-reliability engineering for AGI safety?" by Steven Byrnes
Published 11 hours ago
Description
This post is partly a belated response to Joshua Achiam, currently OpenAI's Head of Mission Alignment:
If we adopt safety best practices that are common in other professional engineering fields, we'll get there … I consider myself one of the x-risk people, though I agree that most of them would reject my view on how to prevent it. I think the wholesale rejection of safety best practices from other fields is one of the dumbest mistakes that a group of otherwise very smart people has ever made. —Joshua Achiam on Twitter, 2021
“We just have to sit down and actually write a damn specification, even if it's like pulling teeth. It's the most important thing we could possibly do," said almost no one in the field of AGI alignment, sadly. … I'm picturing hundreds of pages of documentation describing, for various application areas, specific behaviors and acceptable error tolerances … —Joshua Achiam on Twitter (partly talking to me), 2022
As a proud member of the group of “otherwise very smart people” making “one of the dumbest mistakes”, I will explain why I don’t think it's a mistake. (Indeed, since 2022, some “x-risk people” have started working towards these kinds [...]
---
Outline:
(01:46) 1. My qualifications (such as they are)
(02:57) 2. High-reliability engineering in brief
(06:02) 3. Is any of this applicable to AGI safety?
(06:08) 3.1. In one sense, no, obviously not
(09:49) 3.2. In a different sense, yes, at least I sure as heck hope so eventually
(12:24) 4. Optional bonus section: Possible objections & responses
---
First published:
February 2nd, 2026
Source:
https://www.lesswrong.com/posts/hiiguxJ2EtfSzAevj/are-there-lessons-from-high-reliability-engineering-for-agi
---
Narrated by TYPE III AUDIO.
---
If we adopt safety best practices that are common in other professional engineering fields, we'll get there … I consider myself one of the x-risk people, though I agree that most of them would reject my view on how to prevent it. I think the wholesale rejection of safety best practices from other fields is one of the dumbest mistakes that a group of otherwise very smart people has ever made. —Joshua Achiam on Twitter, 2021
“We just have to sit down and actually write a damn specification, even if it's like pulling teeth. It's the most important thing we could possibly do," said almost no one in the field of AGI alignment, sadly. … I'm picturing hundreds of pages of documentation describing, for various application areas, specific behaviors and acceptable error tolerances … —Joshua Achiam on Twitter (partly talking to me), 2022
As a proud member of the group of “otherwise very smart people” making “one of the dumbest mistakes”, I will explain why I don’t think it's a mistake. (Indeed, since 2022, some “x-risk people” have started working towards these kinds [...]
---
Outline:
(01:46) 1. My qualifications (such as they are)
(02:57) 2. High-reliability engineering in brief
(06:02) 3. Is any of this applicable to AGI safety?
(06:08) 3.1. In one sense, no, obviously not
(09:49) 3.2. In a different sense, yes, at least I sure as heck hope so eventually
(12:24) 4. Optional bonus section: Possible objections & responses
---
First published:
February 2nd, 2026
Source:
https://www.lesswrong.com/posts/hiiguxJ2EtfSzAevj/are-there-lessons-from-high-reliability-engineering-for-agi
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.