Episode Details
Back to Episodes
Your AI Agent Doesn't Know What Its Code Does
Description
You trust your AI coding agent to understand what its code does once it runs. A new benchmark says that trust is misplaced most of the time.
Researchers built SWE-Flux to test it: 480 questions across 12 real Python repos, every answer checked against an actual test run. Not a guess.
The questions aren't "what does this code do." They're "what happens when it runs." Which value moves where. Which branch a loop takes. Why an exception fires.
The best of five models got 37% right. Not on the hardest questions. Overall.
Give them one function. A simple loop, a local check. They're fine.
Ask them to track state across several functions at once, and they fall apart.
More on this in this week's episode of The Human in the Loop.
#AI #CodeReview #TheHumanInTheLoop