November 3, 2026
A language model that gives a different answer to the same prompt is not broken. It is doing exactly what the math underneath it does: run probabilistic inference across parallel floating-point hardware, where summing order can shift between runs and tip a close call between two tokens. Setting temperature to zero does not fix this. It only hides it, which is worse, because it invites trust a system never promised to earn.
The Details:
The same probabilistic machinery that produces variable output also lets a model infer a half-specified request or generate an approach a rigid system would never try. You cannot strip out the variability and keep the reasoning. They are one mechanism, so teams that chase determinism with stricter prompts and silent validators end up protecting a guarantee the model never made.
The fix is a boundary, not a setting. Ask at each step whether a decision must be repeatable and auditable. If yes, let the model propose options, three filenames, three error messages, three refactor paths, and hand the final choice to deterministic code you can read and test. Variability stays in the option set, not the commit.
Prompts need to be treated as build artifacts: versioned, assembled from typed components, logged alongside raw model output. Without that, a drifting system becomes a moving target debugged against another moving target, and "the AI got weird" never turns into a fixable bug report.
Testing shifts from exact-match assertions to an eval harness that tracks pass rate across many runs and classifies failures as safe or catastrophic. A system that fails five percent of the time in predictable ways beats one that fails one percent of the time in ways nobody expected, because a model's failures are quiet. It returns a plausible wrong answer and the pipeline carries on.
Bottom Line: Reliability does not come from taming the model. It comes from the schema, the fallback logic, and the eval harness built around it.
Enjoy this article?
Listen to the Claude Code Conversations radio show or join the community.
