An agent that reports success is not lying. It is reading a real signal and drawing the wrong conclusion from it, and that gap is far harder to catch than a made-up API call.
The Details:
Factual hallucination invents things that do not exist, a function name, a flag, a signature. Reality pushes back fast: the code fails to run, the call errors out. State hallucination cites something real, like an exit code of zero, and overreads it as proof. Exit zero only means the process did not crash; it says nothing about whether the test suite actually ran, or whether it silently collected zero cases.
The structural flaw is that the same agent doing the work also grades the work. No human team would let an engineer sign off on their own deploy with zero review, yet a single continuous chat thread creates the illusion that oversight happened. The conversational pressure pushes toward a closed, confident summary because that is what ends the task, not toward flagging uncertainty.
Absence of errors is not evidence of correctness. Silence in a log can mean nothing broke, or it can mean logging was misconfigured, a code path was never exercised, or an exception got swallowed three layers down. An agent cannot report on what it never saw, and it has no way to flag that blind spot.
The fix is structural, not a smarter prompt. Require the raw artifact behind every claim, the actual test count and output, not the summary. Separate the actor from the auditor with a second pass built to refute rather than confirm. Teams doing this catch cases where a report of all tests passing hides a collected count that quietly dropped from hundreds to a dozen.
Bottom Line: Trust the artifact, not the summary; the summary is a translation, and translations lose exactly the details that matter.
Enjoy this article?
Listen to the Claude Code Conversations radio show or join the community.
