Originally published on Loop & Retry — field notes on building LLM agents that survive production.
The retry-budget post answered the HOW: a shared bucket that caps retries as a fraction of throughput. This post answers the WHEN: when should a call fail immediately instead of burning that budget? The answer depends on what fails, who's waiting, and what happens next. It's the decision layer on top of the budget architecture that prevents the kind of bill you get when those decisions are layered without bounds.
Most code doesn't ask those questions. It retries by default — usually some combination of "well, it might work next time" and "the library gives me retry for free so why not." The answer is: because your retry-ready code becomes your customer's timeout, which becomes your platform's cascade.
User-facing calls: cost of latency beats cost of failure
A request from a user interface has a hard deadline. Your browser won't wait longer than ~30 seconds before showing a spinner forever and your customer closes the tab. Within that window you're choosing between two bad outcomes: a snappy error message or a long hang followed by timeout.
For user-facing operations the cost model is simple:
Cost of one retry attempt ≈ seconds_added_to_response_time × 1
Cost of failing now ≈ user_abandonment_rate × minutes_of_lost_engagement × revenue_per_engagement
In practice, most user-facing services find that a 30-second timeout with two quick retries (1s, 3s backoff) is faster to abandon than a 60-second retry-heavy slog. Amazon measured their own checkout and found each 100ms of added latency costs roughly 0.1% of sales. If your retry attempts add 5 seconds to checkout, that's a direct 5% revenue cut — far larger than the probability that a second attempt would succeed at an error that's already firing on the first.
The decision rule: fail fast on user-facing operations unless the error is known-transient and very rare (network timeout, temporary 503). If it's a real error (bad input, rate limit, service misconfigured), retrying won't fix it — it'll just hang your customer.
Where this lives: at the user request handler, not in library code. If your API client retries in userspace, the handler sees the 3rd attempt as a fresh call and can't do anything about it. Retry logic in the wrong layer makes the decision for the caller.
Background jobs: cost of cascade beats cost of wait
A background job has no impatient user; it has a queue and a deadline hours or days away. Here the tradeoff flips: waiting a few seconds to retry is cheap, but failing and re-queueing means the job supervisor will retry the entire task (not just the failed call). That retry at the job level is the expensive one — it re-runs work that might have been halfway done.
A job that retries internally (the call-level budget) can consume partial progress. A job that gives up and re-queues (the job-level retry) starts over. So background jobs should retry longer than user-facing code — but not infinitely.
Cost of call-level retry ≈ seconds_of_work × (already_sunk_cost_fraction)
Cost of job-level re-queue ≈ seconds_of_entire_job × requeue_overhead
If a job is 10 minutes long and a transient error happens at minute 8, one call-level retry that adds 5 seconds costs 5 seconds. Giving up costs 10 minutes of re-queued work (times the overhead). So retry harder.
But "harder" has a ceiling: if the error isn't transient (the service is really down, not just slow), 100 retries over 10 minutes won't help — they'll just pile up in the job queue behind this one, blocking all the others. The decision is not about trying more, it's about trying faster up to a deadline, then escalating.
The pattern: retry the call with a short backoff (1s, 2s, 4s) up to a total budget of 30-60 seconds. If it hasn't worked by then, do NOT give up — escalate to a human alert that says "this batch is stuck, investigate why the service is down." Then the job waits for the alert to be resolved (with exponential backoff, never-ending retry) rather than burning the queue with re-queues.
Where this lives: at the call site within the job, not at the job-queue level. The job-level retry is for when the entire job fails (code bug, bad input). The call-level retry is for transient service issues. They're different concerns.
Fleet-wide cascades: the retry that multiplies
Now stack multiple layers: N concurrent requests to a downstream service, each with its own retry logic. If the service hiccups, all N requests start retrying simultaneously. A service designed for 100 RPS now sees 300 RPS (retries included). It slows down. More requests timeout. More retries. Cascade.
This is why rate-limited retry budgets exist: they turn N independent retry decisions into one shared decision. But shared budgets only work if each layer respects them.
The failure pattern: A service goes down. Your fleet's 1000 concurrent requests each decide independently to retry. Without a shared budget, you now have 2000 or 3000 requests hammering the struggling service. With a shared budget, you have 1050 (the original 1000 plus a small trickle of retries). The budget is what saves the service from cascade.
The cost model: if you don't have a shared retry budget between your layer and the downstream, assume your retries will cascade and multiply. Price accordingly. The four patterns that prevent this cascade — shared budgets, circuit breakers, decorrelated jitter, and dead-letter quarantine — are covered in distributed retry patterns. This is how you avoid the nested retry cap multiplier that turns a bad deploy into a $200 bill.
Here's the decision tree:
Does this call have a shared retry budget upstream?
├─ YES → use it (your individual call respects the fleet-wide cap)
├─ NO →
│ └─ Is the downstream service under my control?
│ ├─ YES → wire a shared budget immediately, or fail fast
│ ├─ NO (third-party API) →
│ └─ How often does it actually fail?
│ ├─ < 1 per 1000 requests → retry with calm backoff (1-2 attempts)
│ ├─ 1-10 per 1000 requests → log and fail fast (let the caller decide)
│ └─ > 10 per 1000 requests → this isn't transient, investigate why
Stack-level placement: who should decide?
The decision of when to retry should live as close as possible to why the call failed. That's usually not the library level.
Library/framework level: Retry if the error is provably transient (connection reset, timeout, temporary DNS failure). If you can't prove it's transient, don't retry — let the caller decide.
Application level: Retry if you know the operation is idempotent and the downstream's transience is acceptable to your caller. User-facing? Retry less. Background job? Retry more.
Orchestration level: Retry if other callers need protection. This is where shared budgets live. A single request handler using its own budget helps that one request; a fleet-wide budget protects the entire system.
Most code gets this backwards. It retries in the library (one-size-fits-all), then the app layer gives up because it thinks it's already tried, and the cascade happens at the layer where no one can see the full picture.
When to fail fast instead
These are the signals that the call should fail immediately instead of retrying:
The error isn't transient. (Bad input, authentication failure, service responding with 400-range status codes.) Retrying changes nothing; you're just adding latency.
The caller can't afford the wait. (User-facing request with 5 seconds left before browser timeout.) One retry attempt might work, but the added latency makes the failure worse than the success.
The downstream is unhealthy. (Five consecutive timeouts in a row, or >10% error rate.) Retrying is no longer gambling on transience — it's hammering a broken service.
You don't have a shared budget with upstream. (No way to coordinate with other callers.) Your individual retries will cascade if everyone retries the same way.
This call isn't idempotent. (You can't guarantee sending it twice is safe.) Retry only if you can dedup via idempotency keys.
Get these five right and your retry logic becomes protective instead of destructive — it buys resilience for the operations that can afford it, and it fails fast for the ones that can't.
If your fleet is cascading: Distributed retry patterns covers the four patterns — shared budgets, circuit breakers, jitter, and dead-letter queues — that bound a fleet-wide blast radius.
Retry budgets are the resource-allocation layer. This post is the decision layer. Together they let you retry safely: you know when to retry (decision layer) and how much to retry (resource layer). Most platforms have neither, which is why retries and cascades are the same thing in most outages. And once you have both in place, measuring whether retries actually succeed tells you if the strategy is working — or if you're just paying for the appearance of resilience.
