Episode Details

A model developer’s own benchmarking shows the cost of answering the hardest questions in its most demanding category rising three to eighteen times a year, even while the price it charges per token keeps falling. That is not a contradiction. It is the fact a deep tech founder setting a three year or five year AI product price needs to understand before signing anything, because it means the single number most founders reach for, the published per token price trend, describes a different product than the one many of them are actually building.

The headline number is real. GPT-4 equivalent language model performance, priced by OpenAI at $30 per million input tokens and $60 per million output tokens at its March 2023 launch, now runs $0.10 to $0.40 per million tokens across comparable 2026 models, a decline exceeding seventy times in three years. Three forces compound to produce that number: graphics processing units that double or triple their throughput per dollar across each generation, serving software that packs far more simultaneous requests onto each chip than the code it replaced, and model architectures that activate a fraction of their total parameters on any given token.

The serving software piece is the least visible and the most mechanically interesting. Early large language model deployments allocated memory for each request’s attention cache in large contiguous blocks sized for a worst case conversation length, wasting an estimated 60 to 80 percent of that memory to fragmentation, the way an old hard drive wastes space between files. A serving engine called vLLM, built at UC Berkeley and released in June 2023, borrowed the operating system technique of paging memory into small, non contiguous blocks, cutting that waste to under 4 percent and letting a single graphics processing unit hold the cache for far more requests at once. The project’s own published benchmark claimed up to 24 times higher throughput than the Hugging Face library it replaced, and one of its first production users, a research group serving a large public chatbot benchmarking site, reported cutting its graphics processing unit fleet in half after adopting it. NVIDIA’s TensorRT LLM engine, running a quantized Mixtral 8x7B model on a single graphics processing unit instead of two, produced a comparable result: half the hardware, with output quality degraded by only 0.08 percent.

The architecture piece came from a different direction entirely. DeepSeek V3, the model released in December 2024, activates only 37 billion of its 671 billion total parameters on any given token, a mixture of experts design that routes each token through a handful of specialized subnetworks rather than through the entire model. The methodology paper DeepSeek published in May 2025 credits that routing architecture, plus eight bit training precision and a key value cache compressed to roughly one seventh the size of comparable dense models, for training the model on 2,048 graphics processing units, a fraction of the tens of thousands a comparably capable dense model has historically required.

Here is where the story most founders tell themselves stops matching the data. Benchmarking work covering ninety three models found the price for a fixed level of capability falling 31 times a year among the highest capability models and only 1.7 times a year among the lowest capability tier. That gap alone should make any founder suspicious of a single blended number. But the same research found something sharper: the actual cost of achieving frontier level performance, not the per token price but the total dollars spent to get a frontier grade answer, rose three to eighteen times a year over the same period, because each new flagship model reasons longer to answer a hard question, consuming more output tokens, not fewer, even as its price per token keeps falling. Roughly half of the headline capability gain measured on one widely used hard reasoning benchm