Originally published on Loop & Retry — field notes on building LLM agents that survive production.
An agent I was running burned roughly $200 overnight retrying an HTTP 400 — a bad request, the one status code that means "sending this again will fail in exactly the same way." Nothing crashed. No component was individually buggy. Every layer did what it was told: it saw an error and retried, politely, with backoff. The bill was the sum of a dozen reasonable local decisions with no one bounding the total.
This is the incident teardown. The root cause is one idea — retryability is a property of the specific error, not a default you apply to all of them — and the reason it got expensive is a second one: nested retry caps multiply, and a per-step cap bounds a step, never a run and never a fleet. The retry-budgets post worked out the cost of retrying transient failures in detail, modeling how recovery architecture controls the multiplier. This is what happens when the failure isn't transient and you retry it anyway — when the architecture assumes retrying is always safe.
The incident
The agent was an overnight batch job: pull a queue of records, and for each one call an internal enrichment API (a tool) that adds a few fields. It had run clean for weeks. That evening the enrichment service shipped a schema change — one field went from optional to required-and-typed. Records that omitted it now got a 400 Bad Request instead of a 200.
About 40% of the queue was missing that field. So 40% of the calls started returning a 400 that would never stop returning a 400, because the request itself was now malformed. The agent's response to an error was to retry it.
The timeline, reconstructed from logs:
- ~01:00 — the poisoned records start hitting the tool. Each one begins retrying.
- 01:00–07:00 — twelve workers grind in parallel, each one stuck re-attempting doomed calls, each attempt separated by a polite exponential backoff so the storm is slow rather than instant. Nobody's awake. No alert fires, because the only cost alarm was a daily aggregate evaluated at 9am.
- ~08:50 — I check the provider dashboard for an unrelated reason and see the day's spend line already vertical.
- ~09:00 — kill the job. Final damage: $198 and change, and the queue wasn't even finished. Title rounds up.
Zero records were enriched by any of that spend. It was pure waste, and it was waste the system was designed to produce given a permanent error.
Root cause: a 400 is not a 500
Here is the retry wrapper, lightly anonymized. Read it and the bug is right there in the except.
import time
def call_tool_buggy(client, payload, retries=5, backoff=1.0):
for attempt in range(retries):
try:
r = client.post("/v1/enrich", json=payload)
r.raise_for_status()
return r.json()
except Exception as e: # <-- every error is treated as retryable
if attempt == retries - 1:
raise
time.sleep(backoff * 2 ** attempt) # backoff makes the storm last hours
raise_for_status() turns any 4xx or 5xx into an exception, and except Exception swallows all of them identically. But a 500, a timeout, and a 400 are three different claims about the world:
- A 500 or a timeout says the server failed to handle a valid request. Transient. Retrying is correct — the next attempt might land on a healthy replica.
-
A 429 says slow down. Retryable, but only if you honor
Retry-After. - *A 400 (or 404, 422) says the request is wrong. Deterministic. The server evaluated your input and rejected it. Sending byte-identical input again is defined to produce the identical rejection. Retrying it isn't optimistic; it's a guarantee of wasted spend.
The default was backwards. The wrapper treated retry as the rule and success as the exception, when for client errors the opposite holds. The fix is to make the retryable set an allowlist and everything else terminal:
import time, httpx
RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}
class TerminalToolError(Exception):
"""The request is wrong; retrying it will fail identically. Do not retry."""
def call_tool(client, payload, retries=5, backoff=1.0):
for attempt in range(retries):
try:
r = client.post("/v1/enrich", json=payload)
r.raise_for_status()
return r.json()
except httpx.HTTPStatusError as e:
status = e.response.status_code
if status not in RETRYABLE_STATUS:
raise TerminalToolError(f"{status}: {e.response.text[:200]}") from e
if attempt == retries - 1:
raise
wait = _retry_after(e.response) or backoff * 2 ** attempt
time.sleep(wait)
except (httpx.TimeoutException, httpx.TransportError):
if attempt == retries - 1:
raise
time.sleep(backoff * 2 ** attempt) # transport failures *are* transient
Default-terminal, not default-retry. A 400 now fails on the first attempt instead of the fifth. That one change would have cut the tool-layer waste by 5×. It was not, by itself, enough — because the tool layer wasn't the only thing retrying. This is a concrete example of the distributed retry patterns problem: nested caps multiply when there's no upper bound on the total.
Why it got expensive: nested retries multiply
The tool wrapper retries 5 times. But it sits inside an agent, and the agent has its own recovery behavior: when a tool returns an error, the model re-plans and tries again — that's the whole point of an agentic loop. And the agent sits inside an orchestrator, which re-queues a failed item. Three independent layers, each with a cap that looks sane in isolation:
tool_retries = 5 # inside the HTTP client
model_retries = 5 # the agent re-plans on a tool error (cap 4 + the first try)
job_retries = 3 # orchestration re-queues a failed item
attempts_per_item = tool_retries * model_retries * job_retries
print(attempts_per_item) # 75
Seventy-five doomed HTTP attempts for a single poisoned record — none of which could ever succeed, because the input never changed between them. This is retry amplification, and it's insidious because no single number is alarming. Five is a reasonable tool retry. Letting the model try a few approaches is reasonable. Re-queuing a failed job a couple of times is reasonable. You multiply three reasonable numbers and get an unreasonable one, and nobody wrote 75 anywhere for a reviewer to flinch at.
The model re-plans are the part that actually costs money — each one is a fresh call that re-reads the transcript. Tool retries only cost latency (which is why the job ran for hours, not seconds). So the model calls per item are model_retries × job_retries = 15, and each stuck call re-reads a growing context and emits a few hundred tokens of "let me try that differently":
IN_PRICE, OUT_PRICE = 3.0 / 1e6, 15.0 / 1e6 # Sonnet-4.6-class $/token
CTX, OUT = 6000, 400 # tokens re-read / emitted per stuck call
call_cost = CTX * IN_PRICE + OUT * OUT_PRICE # $0.024
model_calls = 15 # per poisoned item
per_item = model_calls * call_cost # $0.36
poisoned_items = 550 # what the fleet chewed before 09:00
print(f"${per_item * poisoned_items:.0f}") # $198
Two of those 15 model calls would have been defensible — the model can't know a priori that the input is bad, so one re-plan attempt is fair. The other 13 were the system re-litigating a settled question. And the model's re-plans made it worse in a way the loop-drift post describes: each attempt appended its reasoning and the 400 to the context, so the window filled with a growing pile of failures, and the model — reading that pile — became more convinced it should keep trying variants. A poisoned input doesn't just cost the retries; it contaminates the window that decides the next retry.
Why the caps didn't save us: local vs global
"But there were caps" — yes, and they all held. No tool call exceeded 5 retries. No agent step exceeded its model-retry cap. Every worker stayed inside every limit it had. The caps were the wrong scope.
A per-step retry cap bounds one step of one run on one worker. It says nothing about:
- The run. 550 items each within cap still sum to $198.
- The fleet. Twelve workers in parallel, each individually legal, were doing ~48 in-flight doomed retries at any given moment. Concurrency multiplies the bill while every worker's dashboard stays green.
- Wall-clock. Backoff, meant to be polite, stretched the storm across seven hours — long enough for a daily-aggregate spend alarm to be exactly useless.
Local caps are necessary (without a per-step cap, one stuck step outspends a thousand healthy runs), but they are not sufficient. You need a limit whose scope is the blast radius: a circuit breaker on the tool and a spend ceiling on the run. A breaker is cheap — it watches the recent error rate and, when a tool starts failing wholesale, stops calling it instead of dutifully backing off into every one of its 75 attempts:
class ToolBreaker:
"""Open the circuit when a tool is failing wholesale — a wall of identical
400s is exactly that signal — so we stop hammering a dead endpoint."""
def __init__(self, window=20, trip_rate=0.5, cooldown=300):
self.recent, self.window = [], window
self.trip_rate, self.cooldown = trip_rate, cooldown
self.open_until = 0.0
def allow(self, now): return now >= self.open_until
def record(self, ok, now):
self.recent = (self.recent + [ok])[-self.window:]
if len(self.recent) == self.window and \
self.recent.count(False) / self.window >= self.trip_rate:
self.open_until = now + self.cooldown # fail fast for 5 min, then probe
With this in front of the tool, the storm ends after ~20 failures, not 8,250. The breaker sees a 100%-failure endpoint and refuses to keep feeding it, and every request while it's open fails fast and terminal instead of grinding through backoff. Twenty wasted calls is a rounding error; that's the difference between an alert and an invoice.
The spend ceiling is the backstop for everything the breaker doesn't catch: a hard, near-real-time cap on run and account spend that kills the job and pages a human, evaluated continuously, not once a day at 9am. If the breaker is the smart limit, the spend ceiling is the dumb one you keep because smart limits have bugs too.
What I'd do
The one-line fix stops this specific incident; the rest stops the class of it.
- Classify before you retry. The retryable set is an allowlist — timeouts, transport errors, 408/429/5xx. Everything else is terminal. Default-terminal, not default-retry. This is the two-line change and it's the highest-leverage one.
-
Make terminal errors terminal all the way up. A
TerminalToolErrormust fail the step, not invite the model to re-plan the identical call. Put the verdict in the error surface — "the input is invalid, do not retry, escalate" — so the model reads it as a stop sign. (Designing errors a model recovers from correctly is its own post.) - Budget your retry amplification. Multiply the caps at every layer — tool × model × job. If the product is 75, that's your worst-case doomed attempts per item. Write the number down and decide, on purpose, whether you can afford it. Usually the fix is to not stack independent retries: retry at one layer, propagate terminally through the others. And build observability into your retry decisions — idempotency keys and correlation IDs let you trace which attempts are part of which logical operation, so you can actually measure whether your per-layer caps are composing as you think.
- Add a circuit breaker and a real spend ceiling. Local caps bound a step; an error-rate breaker and a run/account spend cap bound the blast radius. Evaluate them in near-real-time. A daily cost alarm cannot stop an overnight fire. (The four patterns that scale this to a fleet of workers — shared budgets, breakers, jitter, and dead-letter queues — are covered separately.)
-
Honor
Retry-After, and know what backoff buys you. Backoff makes a storm slower, not smaller — it does not end the storm, it just makes it last long enough to happen while you're asleep. The breaker ends it; the backoff only paces it.
The failure here wasn't a bug in any function. It was a missing sentence in the design: some errors are not worth trying twice, and no local limit knows what the whole system is spending. Retrying is a bet that the next attempt differs from the last. A 400 is the one case where you already know it won't.
What happens at scale: This incident was twelve workers. When thousands of workers each retry independently, the result isn't 12× worse — it's a cascade. Distributed retry patterns covers the four patterns that actually bound a fleet-wide blast radius. And if you're running multi-agent crews instead of distributed workers, a single agent's bad retry can poison the inputs that other agents in the crew reason from.
The numbers here are a reconstruction, not a benchmark: the token sizes, the 550-item figure, and the $198 are a self-consistent model of a real incident, stated so you can swap in your own. The prices are Anthropic's Sonnet-class rates at the time of writing; the lessons — classify before retrying, multiply your caps, bound the global blast radius — are provider-independent.
