Originally published on Loop & Retry — field notes on building LLM agents that survive production.
The retry budget I've described so far treats every retryable error the same way: withdraw a token, back off, try again, and let the shared bucket decide when to stop. That's the right model for a 500 or a timeout — a signal that something broke. It's the wrong model for a 429, which isn't a signal that anything broke at all. A rate limit means the request was received, understood, and refused for exactly one reason: you asked too fast. Lumping it into the same bucket as real failures gets the response wrong in both directions. This separation is especially critical in fleet-wide patterns where a single misclassified signal can trigger a cascade.
Two errors that share a status code family and nothing else
A 500 or a connection timeout tells you the system is in a bad state and might recover if you wait. You don't know how long, so exponential backoff — guess short, double the guess, add jitter — is the right tool for genuine uncertainty about recovery time.
A 429 tells you the system is in a fine state, running exactly as designed, and the problem is entirely on your side of the interaction: you exceeded a quota. Critically, the server usually already knows precisely when you'll be allowed back in, and most APIs that return 429 tell you outright:
HTTP/1.1 429 Too Many Requests
Retry-After: 17
That's not a hint. It's the answer to the exact question your backoff curve is trying to guess. Every major LLM provider's API returns some form of this — Retry-After seconds, or a x-ratelimit-reset timestamp, or both — and the majority of retry wrappers I've read discard it, because the except clause already routes every non-2xx response into one generic retry_with_backoff(attempt) call that only knows the attempt number, not the response.
# common, and wrong for the 429 case
except (Timeout, HTTPError) as e:
if attempt >= MAX_ATTEMPTS or not budget.allow_retry():
raise
time.sleep(backoff_with_jitter(attempt)) # guessing, when the answer was in the response
If the server said "come back in 17 seconds" and your jittered exponential guess says "come back in 1.8 seconds," you retry into the same quota window and get a second 429 — which withdraws a second token from the budget, for a wait you could have avoided by reading a header you already had.
Why one shared bucket makes the wrong tradeoff twice
The point of a shared retry budget is to give a circuit breaker a real signal: when the bucket drains, something is actually degraded and the fleet should back off hard, maybe trip a breaker, maybe page someone. That signal only means something if the thing draining the bucket is evidence of degradation.
A burst of 429s isn't evidence of degradation — it's evidence of success. You're generating enough legitimate traffic to hit a quota. If those 429s draw from the same bucket as your 500s and timeouts, three bad things happen at once:
- The budget drains for the wrong reason. A traffic spike that's entirely healthy (more users, more work, nothing broken) looks identical, from the bucket's point of view, to a backend that's falling over. Your circuit breaker can't tell "we're popular" from "we're failing."
- A real outage hides behind rate-limit noise. If 429s and 500s share a withdrawal count, a spike in legitimate throttling can exhaust the budget before a handful of genuine 500s get a chance to trip anything — the signal you actually built the breaker to catch gets buried under noise it was never meant to detect.
-
You under-wait on the one error where you were handed the exact answer. Guessing a backoff for a 429 when
Retry-Afterwas sitting in the response is strictly worse than reading it, and it costs you an extra request-response round trip against a quota that isn't going to move until the window resets regardless of how politely you ask.
Two buckets, not one
The fix is small: split the withdrawal, not the retry loop. Keep one RetryBudget for genuine failures (timeouts, 5xx, malformed responses — the things a circuit breaker should watch), and give rate limits a separate, much simpler gate that just honors the wait the server told you to take.
class RateLimitGate:
"""Not a budget — there's nothing to ration. Just remembers when you're allowed back."""
def __init__(self):
self.blocked_until = 0.0
def note_429(self, retry_after: float | None, default: float = 5.0):
wait = retry_after if retry_after is not None else default
self.blocked_until = max(self.blocked_until, time.monotonic() + wait)
def wait_if_needed(self):
remaining = self.blocked_until - time.monotonic()
if remaining > 0:
time.sleep(remaining)
def call_model(payload):
for attempt in range(5):
rate_gate.wait_if_needed()
resp = _do(payload)
if resp.status == 429:
rate_gate.note_429(parse_retry_after(resp.headers))
continue # does NOT touch failure_budget — this isn't a failure
if resp.status >= 500 or resp.status is None:
if attempt == 4 or not failure_budget.allow_retry():
raise TransientError(resp)
time.sleep(backoff_with_jitter(attempt))
continue
failure_budget.on_success()
return resp
parse_retry_after is the unglamorous part that actually matters: read Retry-After as seconds or an HTTP-date, fall back to x-ratelimit-reset if that's what the provider sends instead, and only fall back to a guessed default when the response gives you nothing at all. That function is provider-specific glue, but it's the only piece of this that is — the two-bucket split above it is the same shape regardless of which model API you're calling.
The result: rate limits wait exactly as long as they need to and never touch the breaker that's watching for real degradation, and real degradation stops competing with quota noise for the same alarm. Two different problems, worth two different counters — the same lesson as giving the retry budget shared state in the first place, just one layer up. And measuring whether each category actually succeeds helps you tune both: rate limits should succeed after the wait window resets, while genuine failures often need deeper fixes.
