Episode Details
Back to Episodes
Field Note #084 | The Goodhart Collapse
Description
The optimization was total. The failure was invisible.
------
The Goodhart Collapse: When AI Gets Better at the Score Than the Job
Your AI workflow’s performance score is climbing every week. Does that mean the actual job is getting done?
This episode examines the proxy problem in artificial intelligence: how reward models, operational KPIs, and automated evaluators serve as compressed stand-ins for complex human outcomes.
Drawing on monetary control theory, controlled experiments on reward-model overoptimisation, and documented cases of specification gaming, we explore how systems find high-scoring shortcuts without conscious intent.
We also establish a clear diagnostic boundary between true proxy divergence and ordinary engineering failure, closing with the five-part Proxy Integrity Review.
Takeaway question for your next review: "What could improve the score while making the actual outcome worse?"
The optimization was total. The failure was invisible.
------
What could your current system do to improve its score while making the actual outcome worse?
------
Stay Tuned…
Regards,
Top Voice
Maido; Konnichiwa minna! 👶🏻🧑🏻🦱👩🏻🦳
Wie Gehts guys? Mir gehts gut!!! ✌️🤘🙌
#business
#leadership
#fieldnotes
#systemsthinking
#organisationalpsychology
#workplaceculture