Episode Details

Your JSON messages don't reach the GPU as JSON. Somewhere in between, a Jinja chat template flattens them into a string of reserved special tokens — the exact format each model family was trained on. Llama 3 brackets roles with start_header and end_header; OpenAI's ChatML embedded the role in a single opening token; gpt-oss uses Harmony with separate stop tokens for "done talking" and "calling a tool"; Gemma has no system role at all; DeepSeek uses full-width CJK pipes. This episode walks through what special tokens actually are, how tokenizers register them, why the add_generation_prompt flag matters more than anyone thinks, and why a format mismatch never throws an error — it just produces a model that's a bit dimmer than it should be. Key sources: • Hugging Face, Chat templates — https://huggingface.co/docs/transformers/chat_templating • Matthew Carrigan / HF, Chat Templates: An End to the Silent Performance Killer, 2023-10-03 — https://huggingface.co/blog/chat-templates • OpenAI, Harmony Response Format, 2025-08-05 — https://cookbook.openai.com/articles/openai-harmony Episode 6021 — myweirdprompts.com/n/6021 (permanent short link: myweirdprompts.com/765230)