Episode Details

How could all these benchmarks be fake, it's a mystery

https://pivot-to-ai.com/2025/11/06/oxford-pretends-ai-benchmarks-are-science-not-marketing/ - text

• Patreon: https://www.patreon.com/davidgerard
• Ko-Fi: https://ko-fi.com/A1529D5
• Buy me nice things: https://www.amazon.co.uk/hz/wishlist/ls/3Q8VZW46J6DM6
• Get an extremely cool Pivot to AI shirt or mug: https://pivot-to-ai.redbubble.com

Send in your story tips: dgerard@gmail.com

Sources:

• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (press release) https://oxrml.com/measuring-what-matters/
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (PDF) https://openreview.net/pdf?id=mdA5lVvNcU

Previously on Pivot to AI:

• OpenAI o3 beats FrontierMath — because OpenAI funded the test and had access to the questions https://pivot-to-ai.com/2025/01/20/openai-o3-beats-frontiermath-because-openai-funded-the-test-and-had-access-to-questions/
• AI benchmarks are self-promoting trash — but regulators keep using them https://pivot-to-ai.com/2025/02/25/ai-benchmarks-are-self-promoting-trash-but-regulators-keep-using-them/
• The finance press finally starts talking about the 'AI bubble' https://pivot-to-ai.com/2025/09/28/the-finance-press-finally-starts-talking-about-the-ai-bubble/
• video: https://www.youtube.com/watch?v=AgR1TCllRgc&list=UU9rJrMVgcXTfa8xuMnbhAEA

Full Pivot to AI playlist: https://www.youtube.com/playlist?list=UU9rJrMVgcXTfa8xuMnbhAEA