Episode Details
Back to Episodes
What a Cross-Entropy Loss of 1.8 Actually Means
Episode 6502
Published 1 week, 1 day ago
Description
Training runs live and die by cross-entropy loss curves, yet few can translate a single loss number into plain terms. This breakdown walks through how a loss of 1.8 nats translates directly into an average perplexity of six branching choices, why language modeling mathematically equates to compression, and how differing tokenizers produce deceptive per-token numbers. It also explores why tools like EleutherAI's evaluation harness and Karpathy's nanochat bypass token perplexity entirely in favor of bits per byte.
Episode 6502 — myweirdprompts.com/n/6502 (permanent short link: myweirdprompts.com/130615)
Listen on Android: MWP FM Player — myweirdprompts.com/apk