Episode Details

Most people building "AI servers" end up with a machine trying to be a database, web server, chat frontend, and inference box all at once — and then wonder why it falls over. This episode walks through what it actually takes to build a pure inference server: a machine whose only job is running models, with everything served out over an API. We cover engine selection (vLLM vs SGLang vs Ollama), the multi-model problem, concurrency and batching, VRAM planning, and the orchestration gaps that still exist. If you've been thinking about setting up dedicated inference infrastructure, this is the build spec you've been looking for. Episode #527774 — open it directly at myweirdprompts.com/527774