Episode Details

A leaderboard shows a bolded number: one model beat another. But is that a real improvement or ordinary statistical wobble? This episode digs into what "chance" actually means in AI evaluations — the hypothetical worlds behind a null hypothesis, why repeating the same questions with fresh answers is a different experiment than fresh questions, and why the American Statistical Association had to publicly warn that a p-value does not measure the probability that a hypothesis is true. Along the way: random-guessing baselines, Cohen's kappa, noise in human judgment, and the quiet damage done by selecting the best of a hundred prompts. If you've ever bolded a decimal point and hoped, this one's for you. Key sources: • Anthropic — https://www.anthropic.com/research/statistical-approach-to-model-evals • Adding Error Bars to Evals, arXiv — https://arxiv.org/abs/2411.00640 • Measuring all the noises of LLM Evals, arXiv — https://arxiv.org/abs/2512.21326 Episode 6189 — myweirdprompts.com/n/6189 (permanent short link: myweirdprompts.com/963359)