Episode Details

You fine-tune a Whisper-style model or Parakeet on your own voice, run it twice, and get two word error rates several points apart. Neither number is a bug. This episode pulls apart the three layers of ASR evaluation — the scorer, the normaliser, and the harness — and shows why almost every WER you've ever read is really a statement about someone's regular expressions. We walk through jiwer and the Hugging Face evaluate wrapper, the Whisper English normaliser and the points it buys you, why multilingual normalisation can inflate scores by stripping vowel marks, and why code-switched Hebrew-English dictation has no canonical normaliser at all. Then it's on to the harnesses themselves: the Open ASR Leaderboard repo and its three tracks, NeMo's transcribe-and-evaluate pair, and why your RTFx will never match a published one. Plus a salesman who promises negative word error rates. Episode 6207 — myweirdprompts.com/n/6207 (permanent short link: myweirdprompts.com/829930)