Episode Details

TLDR: This post explains two equilibria which regulate the level of AI safety: The first describes how much resources AI companies are willing to spend on AI safety work due to commercial incentives. The second one is about risk awareness and most notably affects government interventions for safety. Doing safety work similar to what AI companies do usually doesn't shift the equilibria much, whereas other work, like more ambitious safety approaches or policy advocacy, do shift them. Both equilibria have become far more important recently, after the Hugging Face (and similar) incidents.

The equilibrium of commercial safety interests

Consider this simplified model:

• AI companies have commercial incentives to invest in safety research: it improves their brand and prevents their AIs from causing harm that triggers lawsuits or regulation. Therefore they will fund safety work until the marginal commercial benefit of investing a dollar in safety equals the marginal commercial benefit of investing a dollar in AI capabilities.
  • Thus, if you're at an AI company doing commercially-incentivized safety work, e.g. training models to not take harmful actions, the counterfactual impact (henceforth just "impact") of the safety work you produce is roughly zero because it would've been done anyway. [...]

---

Outline:

(00:49) The equilibrium of commercial safety interests

(02:17) FAQ

(04:15) The risk awareness equilibrium

(08:18) How might we want to invest in safety research then?

(11:45) Conclusion

(12:19) Appendix: But isn't there also an equilibrium for policy advocacy?

The original text contained 11 footnotes which were omitted from this narration.

---

First published:
September 5th, 2026

Source:
https://www.lesswrong.com/posts/kxHiSsNh4MH82nhXD/assessing-the-impact-of-safety-work-needs-equilibrium

---

Narrated by TYPE III AUDIO.