Episode Details

Every token an LLM generates has to attend to every token before it, and the KV cache is what keeps that from being recomputed from scratch — while quietly becoming the thing that caps your context length, your batch size, and your latency. This episode walks through what's actually stored (keys and values, never queries), why prefill is compute-bound and decode is memory-bandwidth-bound, and the formula that turns a single Llama 2 conversation into two gigabytes of cache. Then it maps the whole lineage of fixes onto three multiplicative dimensions — entry size, sequence length, and layers — from Multi-Query and Grouped-Query Attention through DeepSeek's Multi-head Latent Attention, DeepSeek Sparse Attention, cross-layer sharing, and the inference-only tricks like PagedAttention and KV quantization that need no retraining at all. Episode 6024 — myweirdprompts.com/n/6024 (permanent short link: myweirdprompts.com/640482)