Why it matters: A test suite full of green checkmarks can hide the exact bugs it was supposed to catch, because AI often writes assertions that just restate what the code already does.
The Details:
When a model generates tests by reading existing code, it encodes current behavior as the expected result. If a loop stops one iteration early, the test asserts that shorter length and calls it correct. The function and the test now agree perfectly, but agreement is not verification. A bug that used to be an open question becomes a closed one, and the next engineer has to fight a passing suite to fix it.
Coverage percentage measures which lines executed, not which failure modes anyone considered. The incidents that actually hurt teams live in the untested edge: the empty list, the negative number, the double form submission, the timezone that only breaks in production. A smaller suite written by someone actively trying to break the system catches more of these than a large one generated from the implementation.
Real test value comes from intent that lives outside the code: the ticket, the spec, the conversation about what the feature was meant to do. AI can't read that unless it's given it. Pasting acceptance criteria or a one-line contract into the prompt before asking for tests, or writing the assertions yourself and letting AI fill in setup and fixtures, keeps judgment where it belongs.
Not every function needs equal scrutiny. The functions touching money, auth, or data mutation deserve human-written assertions. Formatters and getters can take generated scaffolding without much risk, so triage matters more than blanket rigor.
Bottom Line: AI can build the scaffolding around a test, but only a human can decide what correct actually means.
Enjoy this article?
Listen to the Claude Code Conversations radio show or join the community.
