Episode Details

This is a link post.

Thanks to Arepo, David Thorstadt, Zeshen, and Michael st Jules for looking over this article. Disclaimer: I am not a subject matter expert and this is not a rigorous scientific article. This post is entirely human-written.

Introduction

In a previous article, I wrote a general introduction to the optimizer's curse, the phenomenon where a combination of random errors and ranking of causes results in the overestimation of top causes and a biasing of results towards uncertain causes. In that article, I mainly focussed on areas such as global health charities, where estimates or errors can be derived from the results of reasonably strong empirical evidence such as randomised control trials.

In this post, I want to explore how the optimizer's curse might affect tasks that are in the domain of extreme uncertainty. Specifically, I will try and model how the curse could manifest in the ranking of estimates of existential risk, a subject where empirical evidence is thin and experts disagree by orders of magnitude.

A standard way of ranking existential threats is to take a bunch of threats that have been raised as potentially deadly to humanity, and then for each one of them [...]

---

Outline:

(08:09) Part 1: Building a model

(25:19) Part 2: Interrogating the model

(40:15) Part 3: Objections

(57:26) Summary and Conclusion:

---

First published:
June 11th, 2026

Source:
https://forum.effectivealtruism.org/posts/mEJKkkjQydM68EQTP/are-top-existential-risk-estimates-50-000-times-too-high-an

Linkpost URL:
https://titotal.substack.com/p/are-top-existential-risk-estimates

---

Narrated by TYPE III AUDIO.

---