Episode Details

Current AI risk management relies on qualitative approaches, much like nuclear safety before 1975. We propose a shift to quantitative risk modeling, following the approach that transformed nuclear safety. We propose a methodology and demonstrate it by building nine probabilistic models of AI-enabled cyber attacks. This is a first attempt at AI risk quantification. We invite criticism and hope this can be a starting point for the kind of iterative improvement that made nuclear safety robust.

Introduction — why quantitative risk modeling matters

In 1975, the Nuclear Regulatory Commission (NRC) published WASH-1400, the first systematic application of probabilistic methods to nuclear reactor safety. Before this, nuclear reactor safety was grounded in a conservative, qualitative approach. The argument was that consequences of even the worst accident were effectively zero because physical barriers would prevent any radioactive release. But in 1966, experts recognised that a core meltdown in large reactors could breach those barriers. Regulators now had to argue that severe accidents were possible but sufficiently improbable. That required specific probability estimates.

The shift to quantitative methods was not straightforward. When WASH-1400 began, one technical advisor wondered, "Do we dare undertake such a study till we really know how?" The 1975 [...]

---

Outline:

(00:46) Introduction -- why quantitative risk modeling matters

(04:51) Our methodology

(13:15) Initial insights from quantitative AI risk modeling

(13:58) 1. Risk increases as AI capabilities increase. So does uncertainty.

(15:15) 2. Different benchmarks capture risk differently.

(16:24) 3. Attack bottlenecks change as capabilities change.

(18:15) 4. Factors driving risk from current AI systems are different from factors that may drive risk in the future.

(19:23) Open problems

(22:09) What's next for SaferAI

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
May 27th, 2026

Source:
https://www.lesswrong.com/posts/CpmpyS9Rpz7obejaw/quantitative-ai-risk-assessment-a-starting-point-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Listen Now