Episode Details

Mistral Large 4 weights, due in late October 2026, need about 1.05 terabytes of GPU memory at 8 bit precision, so switching from the API is an engineering decision. Mistral released the model on October 6, 2026 with 1.05 trillion total parameters, 49 billion active per token, a one million token context window, and prices of $1.36 per million input tokens and $4.18 per million output tokens. A startup that treats the weights release as a price cut will be disappointed. A startup that builds a provider neutral model layer now will treat it as a configuration change.

The arithmetic surprises people. Only 4.7 percent of the parameters are active for any token, which makes compute cheap. The router, though, can send any token to any expert, so every expert must sit in memory. At 8 bit that is 1.05 terabytes of weights, more than the 640 gigabytes in an eight GPU H100 node and just under the roughly 1,128 gigabytes in an eight GPU H200 node, with about 78 gigabytes left for the cache that long prompts consume. Quantized to 4 bit, the weights shrink to about 525 gigabytes and fit on the H100 node, with quality loss that nobody has yet measured on this model.

What does self hosting actually cost compared with the API?

Self hosting beats the API price only when a cluster stays busy around the clock. A vendor of rented GPUs lists an eight GPU H200 node at $44.08 per hour on demand. At the 3 to 1 input to output mix many applications run, the blended Mistral price is $2.065 per million tokens, so that node must process about 5,900 tokens per second, every second, to match it. The same vendor’s guidance for the smaller Mistral Large 3 reports about 2,500 tokens per second on that hardware, and its own cost table disagrees with its throughput math by a factor of three. I trust the division over the table.

A redundant production pair at spot prices runs roughly $464,000 per year before any engineer is paid. That buys about 225 billion tokens a year at API rates. Below that volume the API wins on price. Above it, or when a customer’s security addendum forbids third party subprocessors, or when the product depends on a fine tuned variant, the private path earns its cost.

What should the model layer contain?

The layer needs six parts. A normalizer converts every application call into one internal schema, so the product never touches a vendor SDK. A policy router picks a target per task class, such as contract extraction or code review, from configuration that carries a fallback order, a cost cap, a latency budget, and a data class rule. Adapters absorb vendor quirks in tool calls, streaming, and usage fields. An evaluation harness holds frozen test sets drawn from de identified production traffic and runs offline, shadow, and canary comparisons. An observability store records tokens measured by an independent tokenizer, because different tokenizers count the same text differently and a price per million tokens means little until measured on your own traffic. A weights supply chain downloads once, verifies checksums, mirrors to private storage, pins revisions, and loads only safe tensor files.

Prompts must be versioned per model. I have reviewed enough vendor transitions to say that the prompt is the part that breaks first, and it breaks quietly: tool call arguments drift, JSON closes late, refusals change. A regression suite that runs on every target change catches the drift before a customer does.

Which benchmark numbers deserve trust?

Few of them yet. Mistral reports 61.7 percent on DeepSWE v1.1, 93 percent on Cybench, and a human evaluation score of 3.74 out of 5.0 that ranks second of five models. The first independent broad index puts the model at 48.05 percent, 32nd of 44. A legal agent benchmark ranks it sixth of 75 with a 15.83 percent pass rate. Independent testing found effective context nearer 512,000 tokens than one million. Capability is uneven by task, which is the best ar