Episode Details
Back to Episodes
Who Grades the Grader? AI Writing Benchmarks Put to the Test
Episode 6042
Published 1 week, 6 days ago
Description
Every headline quotes maths and code benchmarks, but neither measures what actually matters for long-form writing: whether a model holds its voice across thousands of words, or engages with uncomfortable questions instead of refusing. This episode walks through five specialist leaderboards built to fill that gap — EQ-Bench Creative Writing v3, Longform, UGI, SpeechMap, and PodBench, a new ACL paper scoring podcast scripts specifically — and unpacks the awkward part: where DeepSeek V4.1-Flash, the model writing this very episode, lands on each of them. Along the way: why the most willing model on the board is also one of the worst at long-form, what it means that LLM judges score other models' prose, and how a benchmark stops being a mirror and starts being a selection pressure.
Episode 6042 — myweirdprompts.com/n/6042 (permanent short link: myweirdprompts.com/733948)