A model that passes every staged test can still fail constantly once real users show up, because the gap is not a bug in the code. It is a mismatch between how the system was measured and how it gets used.
The Details:
Test sets are written by the same engineers who built the feature, so they cluster around clean, average-length inputs. Real traffic is a long tail: pasted logs, broken connections, phrasing nobody anticipated. A suite that only covers the middle of that distribution will always look healthier than the system actually is.
Language model output is unusually sensitive to surface phrasing, not just intent. Two users asking the same thing in different words, different punctuation, different capitalization, can get meaningfully different answers. Building paraphrase variants of each test case, five to ten per prompt, and rerunning them multiple times exposes fragility that a single clean test run hides entirely.
Eval suites go stale the moment they are written, because user vocabulary and behavior keep shifting while the test data sits frozen. Teams that treat evaluation as a one-time deliverable are quietly measuring a version of reality that no longer exists months later.
Explicit feedback like thumbs-down clicks captures a tiny fraction of dissatisfaction. Repeated resubmissions, edited outputs, and abruptly ended conversations are the real signal, and ignoring them means regressions go unnoticed until they have already cost trust. Acting on that signal needs deliberate delay too, since reacting to every complaint immediately risks overfitting to a handful of loud users.
Bottom Line: Staging tells you the code runs; production tells you whether the system actually works, and only continuous, adversarial, real-input testing closes that gap.
Enjoy this article?
Listen to the Claude Code Conversations radio show or join the community.
