A model that answers correctly once in a demo tells you nothing about whether it will survive real traffic. The gap between those two states is infrastructure, and most teams only discover it after launch.
The Details:
An LLM call behaves nothing like a normal API request. It can run for a minute or more, and running it inside a synchronous web request means one slow generation backs up your server and invites duplicate calls you still pay for. A job queue fixes this by turning each generation into a durable, retryable unit of work that survives a restart. But retries need a failure taxonomy first: a timeout deserves a retry, a fluent but wrong answer that passed format checks does not, since retrying it just burns money on the same mistake.
AI systems carry two kinds of state that get treated identically and shouldn't. Workflow state, like which step ran, is ordinary database work: idempotent writes, crash recovery. Conversation history and the exact prompt that produced an output are not cache, they are the only record that lets anyone explain a strange result weeks later, so they need to be stored raw and versioned, not summarized.
Human oversight only works when it is a hard boundary in code, not a guideline someone can skip under pressure. The real design question is which actions are reversible. A drafted reply a human edits is low risk; a refund or a production data change issued autonomously is not, and the system should be built so the AI physically cannot complete that second kind alone.
Latency and error-rate dashboards can look perfectly healthy while output quality quietly rots, because hedging, drift, and confidently wrong answers pass every format check. Real observability tracks cost per successful outcome, downstream validation failure rates, and sampled full traces you actually read, not just aggregated numbers.
Bottom Line: The prototype proves the model can do something once; production is the separate discipline of making it do that reliably without anyone watching.
Enjoy this article?
Listen to the Claude Code Conversations radio show or join the community.
