A model that rewrites your code will happily keep every test passing while quietly deleting a rule nobody ever wrote down.
The Details:
Tests only check outputs, not the path taken to reach them. A refactored payment function can still charge the right amount while writing a duplicate record under certain conditions, because no test ever asserted the write happens exactly once. That rule lived as an unwritten invariant in someone's memory, not in the test suite, so the model had no signal telling it the behavior was load-bearing.
Accidental behavior and required behavior look identical in a diff. A sorted list that happens to come back ordered and a sorted list that must stay ordered because a report depends on it are indistinguishable to a tool working purely from code. Humans carry that distinction in institutional memory; AI carries only what's in the prompt or the repo.
The fix is to mine your own incident history. Postmortems containing the phrase "we assumed" are a catalog of invariants that were already protecting production by luck. Rank candidates by blast radius, focusing on money, identity, and time, then add each check in shadow mode, logging violations without throwing errors for a week before converting it to a hard stop.
Encoding alone doesn't stop drift. Invariants need to live where an AI actually reads context, such as a CLAUDE.md file or repo-level documentation, with each rule tied to the incident that revealed it. A rule that only exists in a wiki page is invisible to a model editing the code, so it will not survive the next pass.
Bottom Line: A passing test suite no longer means the system works, it means the system matches whatever you bothered to write down.
Enjoy this article?
Listen to the Claude Code Conversations radio show or join the community.
