Episode Details

You ask a model to write an email, then pause and say, "Actually, is this topic too obscure?" A good model answers the question. A bad one splices it right into the draft. This episode unpacks this surprisingly common failure mode — what cognitive abilities it requires, why it's not being measured, and how to build a rigorous benchmark for it. We explore pragmatic reasoning, discourse parsing, theory of mind, and conversation state management, plus the four levels of failure from direct contamination to frame collapse. Featuring a proposed Multi-Level Conversation Boundary Test (MCBT) that reveals why even GPT-4o and Claude 3.5 Sonnet fail 15-30% of the time.