Episode Details

Hosted batch inference is one of the best deals in AI right now — half price, no latency requirement, just hand over a file. But what if you want that on your own GPU? This episode unpacks why the hosted batch API is really a scheduling and billing construct, and what you actually have to build locally to reproduce it. We walk through continuous batching, the KV cache budget that secretly governs your concurrency, and the two shapes a local batch job can take: offline in-process, or server plus concurrent client. Then we compare the real options — vLLM's offline class and its newer run-batch command, SGLang's prefix-sharing strengths and AMD weak spots, Ollama's development-tool defaults, llama.cpp's portability, and why Hugging Face's text generation inference is no longer a starting point. Episode 5991 — myweirdprompts.com/n/5991 (permanent short link: myweirdprompts.com/694162)