Production breaks are not usually a model problem. They are a specification problem wearing a model's coat. When an AI builds exactly what a clean spec describes, it also builds exactly the gaps that spec left out, and better models only make those gaps harder to spot because the output looks finished.
The Details:
- A tidy spec like "run every fifteen minutes, pull new records, write to the warehouse" never says what happens when run three overlaps run four, or when a crash leaves the cursor in an ambiguous state. That missing context is not a knowledge gap, it is operational scar tissue that only shows up after something pages you at 2am.
- Tests written by the same model that wrote the code inherit the same blind spots. If the implementation assumes a non-empty list, the test feeds it a non-empty list, so the green checkmark proves nothing about reality. Real coverage has to come from outside the code's own assumptions: bug reports, production logs, past incidents.
- Asking an AI to attack its own code, assuming the author was sloppy, produces a completely different test set than asking it to write tests for a function. Same model, different question, different output, because the prompt changes what gets interrogated instead of just confirmed.
- Concurrency, partial failure, and unstated assumptions are the three places specs lie by omission. Asking directly what the AI had to assume before writing any code, or what the system looks like mid-crash rather than before and after, forces those assumptions onto the page where they can be checked.
Bottom Line: Treat the spec as a conversation that admits what worries you, not a contract that hides it, and the AI stops solving the brochure version of your problem.