Our cache cluster blipped for four seconds at 09:14. It came back at 09:14:04 and immediately fell over again, and this time it stayed down for nineteen minutes. Nothing was wrong with the cache. We had DDoS'd ourselves with our own retry policy, from roughly 900 pods running the same library.
The code looked responsible. It was the pattern every blog post recommends:
import random, time
def backoff_delay(attempt, base=0.5, cap=30.0):
# exponential backoff, no jitter
return min(cap, base * (2 ** attempt))
Why synchrony is the bug
Here is the mechanism I missed for a long time. Every client that fails at the same wall-clock instant computes the same next-attempt time, because the delay is a pure function of the attempt number. Pods don't have independent attempt counters in the interesting case — they all fell over together when the dependency degraded, so they're all on attempt 0, then all on attempt 1, and so on.
The result is not a smooth smear of retries. It's a train of discrete spikes. At t+0.5s, all 900 pods retry. The remaining capacity absorbs maybe half of them. The other half fail, and now they all retry together at t+1.5s. Each wave is bigger than the load the dependency can take, and the waves line up exactly at the moments the dependency is weakest — because a saturated service recovers slower than it degrades.
The failure mode has a name worth knowing: a retry storm, or metastable failure. The system doesn't return to its original state when the trigger is removed, because the retries themselves are the load.
I confirmed it by logging not the retry count but the histogram of retry timestamps across the fleet. A healthy client pool produces a flat distribution. Ours had three sharp bars, 500ms apart. That histogram is the measurement you want — it's cheap, and it tells you immediately whether you have a synchrony problem.
Full jitter and decorrelated jitter
The fix is to make the delay a random variable instead of a function. AWS's architecture blog popularized the comparison, and full jitter is the one I use by default:
import random
def sleep_full_jitter(attempt, base=0.5, cap=30.0):
# pick uniformly from [0, capped exponential]
return random.uniform(0, min(cap, base * (2 ** attempt)))
def sleep_decorrelated(attempt, base=0.5, cap=30.0, prev=0.0):
# sleep = min(cap, random(base, prev * 3))
return min(cap, random.uniform(base, prev * 3))
Full jitter spreads arrivals uniformly across the window. Decorrelated jitter is stateful: each delay is drawn between base and three times the previous delay, which gives a random walk that widens over time and tends to keep the average load lower on long outages. Full jitter is simpler and has no per-attempt state to get wrong. Decorrelated wins when retries can last for minutes and you want the tail to stretch.
The important property of both: the expected rate of retries stays roughly constant rather than spiking, even if the number of clients is large. That's the whole point. You are converting a synchronizing process into a dispersing one.
Retry budgets beat per-call counts
Jitter alone wasn't enough for us, because the request amplifier was still unbounded. The real control is a retry budget: cap retries as a fraction of total requests per client, not per call.
Instead of "each call may retry 3 times," say "this client may issue retries equal to 10% of its successful requests, refilled continuously." Google's SRE book describes this as a retry budget; Finagle exposes it as RetryBudget. The property you get is proportionality: when the dependency is healthy, retries are cheap and plentiful. When it's degraded and most requests fail, the budget drains and retries stop almost entirely. It's a negative feedback loop where per-call counts are a fixed multiplier.
A budget also bounds the worst case. With 3 retries per call at 5 layers of the stack, one user request can become 4^5 = 1024 calls. I have seen this exact multiplication in a service mesh, and it is not theoretical.
Circuit breakers and idempotency
Two things have to be true or none of this works.
The circuit breaker trips on failure rate over a rolling window, and it must have a half-open state that admits a small number of probes. Without it, retries keep the load on a dead dependency while the breaker's state machine still reports closed. With it, the breaker and the budget do different jobs: the budget limits how much extra traffic you generate, the breaker stops generating traffic at all.
Idempotency is the harder one. Retrying a non-idempotent POST /payments doesn't amplify load, it duplicates money. Every retried operation needs an idempotency key that the server deduplicates on, and the client needs to send the same key on retry, not a fresh one. If you can't make an operation idempotent, don't retry it — return the error and let the caller decide.
The version of backoff_delay I run today is roughly fifteen lines, and the diff that mattered was adding random.uniform(0, ...). That one call was the difference between a four-second blip and a nineteen-minute outage.
I write about production failures in Postgres, queues, and distributed systems.
