Originally published on Loop & Retry — field notes on building LLM agents that survive production.
The demo works. It always works — that's what a demo is. You gave the agent a representative task, it took the happy path, and it produced the right answer in front of an audience. What the demo proved is that the agent can succeed once, on an input you chose. What it did not tell you is the number you actually need: how often it will fail on the inputs you didn't choose, under load you didn't apply, from users who don't know or care what shape you expected. That gap — between "succeeds once on a good input" and "fails 8% of the time on the real distribution" — is where production failure lives, and most of it is predictable before release if you test the right things.
This is the pre-release companion to measuring agent failure in production. That post is about instrumenting failures once they're live. This one is about forecasting them while you can still cheaply do something — because the cheapest failure to fix is the one you caught in a test harness, and the most expensive is the one your users find. The point isn't to prove the agent works. You know it can. The point is to make it fail on purpose, on your schedule, and read the rate.
What changes between the demo and production
The demo and the deployment run the same code. Four things differ, and each one is a failure source the demo structurally can't show you:
- Input distribution. The demo uses inputs you picked; production uses inputs users bring. The tails you never imagined — empty fields, wrong encodings, a 40-page document where you tested a paragraph, a language you don't support — are where agents fall over, and they're most of the real distribution, not an edge of it.
- Load. The demo runs one task at a time. Production runs many, sharing rate limits, connection pools, and a context budget. Failures that only appear under concurrency — throttling, timeouts, resource exhaustion — are invisible at N=1 by construction.
- Adversarial and malformed input. The demo assumes good faith. Production includes users who paste garbage, injection attempts riding in on tool output, and inputs crafted to break you. None of this shows up unless you supply it.
- Duration and drift. The demo runs for a minute. Production runs for months, across model updates, dependency changes, and upstream schema shifts — the exact class of change that caused the $200 postmortem, where a field going from optional to required turned 40% of a queue poisonous overnight.
Every one of these is testable before release. The reason they usually aren't is that testing them requires deliberately trying to break the thing you just got working, which is psychologically the opposite of what a green demo makes you want to do.
Four signals that forecast production failure
Here's what I run before shipping, ordered by how much production failure each one predicts per hour of effort.
1. Failure injection: force the errors and measure recovery. Don't wait for a tool to time out in production to learn what the agent does. Inject the failure in a harness — make the tool return a 500, a 429, malformed JSON, an empty result, a timeout — and assert on the recovery, not just the happy path.
FAULTS = {
"timeout": lambda: (_ for _ in ()).throw(TimeoutError()),
"http_500": lambda: {"status": 500, "body": "internal error"},
"http_429": lambda: {"status": 429, "retry_after": 30},
"malformed": lambda: {"status": 200, "body": "{not valid json"},
"empty": lambda: {"status": 200, "body": "[]"},
"wrong_schema": lambda: {"status": 200, "body": '{"unexpected": true}'},
}
def probe(agent, fault_name):
"""Run the agent with one fault injected; record what it does."""
result = agent.run(task=REPRESENTATIVE_TASK, tool_fault=FAULTS[fault_name])
return {
"fault": fault_name,
"recovered": result.completed and result.correct,
"retries": result.retry_count, # did it retry a non-retryable error?
"cost": result.total_cost, # did one fault blow the budget?
"escalated": result.asked_for_help, # did it fail loudly or silently?
}
for fault in FAULTS:
print(probe(agent, fault))
The output is a failure-mode matrix: for each fault, did the agent recover, and how much did failing cost? The two rows that matter most are malformed/wrong_schema (does a poisoned tool result cascade, per how failures cascade?) and http_429/http_500 (does it retry a permanent error into a runaway bill?). If the agent retries malformed fifty times or escalates nothing when it gives up, you've just found a production incident in a unit test.
2. Distribution testing: run the real tails, not the mean. Pull a sample of actual historical inputs — or the closest proxy you have — and run the agent across all of them, not the three you'd demo. Sort the results by failure and cost. You're looking for the shape of the tail: what fraction fail, and are the failures concentrated in an input class you can characterize (long documents, a specific language, a field that's often empty)? A characterizable failing class is a fix; a diffuse one is a redesign. Either way the rate on a representative sample is your single best point estimate of the production failure rate, and it's available before you ship.
3. Load testing with the cost model attached. Run the agent at production concurrency against a staging backend and watch two things: the failure rate under contention (does it climb when workers share rate limits?) and the cost per task under retry (does the retry multiplier you budgeted for hold when failures cluster?). Concurrency-induced failures are the ones that make people say "it worked in staging" — because staging ran it once. Run it a thousand times in parallel and the throttling, the pool exhaustion, and the correlated retries all show up.
4. Canary and shadow before full traffic. The honest pre-release test is a small slice of real traffic. Shadow-run the agent against production inputs without acting on the outputs, and compare its results to the incumbent (a human, an old system, a simpler agent). This catches the distribution and load problems the harness approximated, on the genuine article, with the failures contained because nothing downstream consumes the shadow output yet. A canary that fails at 8% when you expected 1% is a launch you're glad you staged.
The signals that don't predict much
Worth naming, because they absorb effort that could go to the four above:
- More happy-path examples. A tenth successful demo run predicts almost nothing a first one didn't. Success on chosen inputs is not evidence about unchosen ones.
- Prompt-level unit tests on ideal inputs. Useful for catching regressions, near-useless for forecasting production failure, because the whole problem is the inputs you didn't write a test for.
- Aggregate benchmark scores. "85% on the eval set" tells you about the eval set. Whether the 15% failures align with your production distribution and your cost tail is the actual question, and the headline number doesn't answer it.
The through-line: predictive tests are the ones that introduce something the demo lacked — a fault, a tail input, concurrency, real traffic. Tests that stay on the happy path, however many you run, just re-prove the demo.
What I'd actually do
- Budget failure-injection time like you budget feature time. The fault matrix from signal 1 is a few hours of work and it finds the retry-a-permanent-error and cascade-on-poison bugs before they have a production bill attached. It's the highest-leverage pre-release test there is.
- Estimate the production failure rate from a real-input sample, and write it down. Not "it works" — a number, with a date, that you can compare against the live rate once instrumentation is up. If the live number is far off your pre-release estimate, your test distribution was wrong, and that's its own useful finding.
- Stage the launch: harness → load → shadow → canary → full. Each stage introduces one more thing the demo hid, with the failures still cheap and contained. Skipping stages doesn't make the failures not happen; it just moves the discovery to production, where the same failure costs the most across every axis. If your system uses multi-agent teams, the modes of failure unique to multi-agent systems — agent disagreement, circular delegation, cascading errors — require their own pre-release tests. A single agent's failure injection tests don't cover crew coordination problems.
The demo proves the agent can succeed. Predicting production failure is the opposite exercise — making it fail deliberately, at low stakes, and reading the rate before your users read it for you. The failures are coming either way. The only choice is whether you meet them in a test harness or on the invoice.
The injection harness above is a sketch of the pattern, not a framework — the recovery assertions and fault set are where the real work is, and they're specific to your tools. The categories (input distribution, load, adversarial, drift) transfer across providers and models.
