Episode Details

A HumanLayer engineering team spent four months last year proving that an AI coding agent left to write, review, and deploy its own code will quietly ruin a codebase well before anyone notices, and the specific reason it happened is the same reason your own agent feature is at risk right now.

Dex Horthy, the chief executive and cofounder of HumanLayer, told The Pragmatic Engineer’s Gergely Orosz in a July 15, 2026 interview that his company ran a fully unattended coding system between July and November 2025, with Claude Opus based agents writing, reviewing, and merging code without a human in the loop. Three months in, a single defect surfaced that took weeks of manual work to trace, because it sat inside layers of agent generated code that no person had ever read. Horthy now calls the experiment a cautionary tale.

The failure was not a bad model. It was a context window nobody was managing.

Every large language model attends to every token in its context window at every step, and that attention mechanism has a documented blind spot. A Stanford team led by Nelson Liu showed back in 2023, in a paper later published in the Transactions of the Association for Computational Linguistics, that models retrieve information near the start or end of a long context far more reliably than information buried in the middle. Chroma Research confirmed the pattern still holds in current generation models in a July 2025 report that tested 18 systems from Anthropic, OpenAI, Google, and Alibaba. Context windows have grown from roughly 4,000 tokens in 2023 to 1,000,000 tokens in some models today, and the blind spot has not closed. It has just moved further out.

Horthy calls the result the dumb zone, and his own number for it is specific: keep active context under roughly 40 percent of a model’s nominal window, which works out to around 300,000 to 400,000 tokens against a 1,000,000 token model and around 100,000 tokens against a smaller one. That figure comes from his team’s own analysis of 100,000 developer coding sessions, not a marketing claim about maximum context length. Push past it and a model’s judgment measurably degrades, even though the window technically still has room.

There is a tell for when that has already happened, and it is almost funny once you know to look for it. Horthy watches for the model saying you’re completely right or you’re right to push back on that in response to ordinary correction. In his telling, that phrase does not mean the model reconsidered anything. It means the session has degraded to the point that the model is agreeing rather than reasoning, what he calls trajectory poisoning. At that point, arguing further inside the same conversation wastes time the team does not get back.

The fix Horthy actually uses is not a better model or a bigger window. It is a discipline he calls intentional compaction: before a session’s context reaches the dumb zone, compress the verified, useful state (the design decisions, the known defects, the plan) into a clean Markdown document, have a human or a fresh session check it, and then start over from that document instead of continuing inside the degraded one. For a complex feature, he runs four separate sessions in sequence: one reads the codebase and writes a research document, one turns a work ticket plus that document into a design document, one turns both documents into an implementation plan, and one executes the plan. Human review concentrates on the design step, because that is where Horthy has found models weakest, not on auditing every line the model eventually writes.

The economics line up with the engineering. Horthy’s own data points to three operating models for AI assisted delivery: the unreviewed version that failed in four months, full human review of every generated line, which he estimates produces a 30 to 50 percent productivity lift over pre AI engineering, and a design focused review model that he reports runs 2