Retry budgets by language: Python, Go, and JavaScript

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

The retry-budget argument is language-independent: your retry multiplier is set by how you recover, not by how often you fail, and a per-call cap bounds a call but never a run or a fleet of workers. That's the theory, grounded in the $200 cost of nested caps that multiplied across layers. The theory doesn't tell you where in a Python decorator, a Go for loop, or a JavaScript promise chain the budget actually lives — and each language makes a different part of it easy to get wrong.

This post is the implementation note. Same budget, three runtimes, three traps.

The one invariant, restated for code

A retry budget is a bucket shared across everything that might retry, refilled slowly, that caps retries as a fraction of throughput rather than as an absolute count per call. Two properties matter and both are easy to lose in translation:

1. The budget is shared state. If every call site owns its own counter, you don't have a budget — you have N caps that multiply.
2. A denied retry is a real outcome. When the bucket is empty the call fails now, deliberately, instead of waiting for a slot. Retrying is a privilege the system can revoke, not a right the call holds.

Here's a minimal bucket, deliberately boring, that all three languages will share the semantics of:

budget = token bucket:
  capacity C, refill R tokens/sec
  on each SUCCESS: deposit ratio_bonus (e.g. +0.1 token)
  on each RETRY:   withdraw 1 token, or DENY if empty

The refill and the success-bonus are what make it a budget: retries are funded by the work that's succeeding. When failures spike and successes stop, the bucket drains and retries stop with it. That's the whole point. Now the three ways to wire it in.

Python: the decorator hides the budget from itself

tenacity is the reflex, and it's good, but the default shape teaches you the wrong lesson:

@retry(stop=stop_after_attempt(4), wait=wait_exponential())
def call_model(payload): ...

That decorator is a per-call cap. Every function it wraps gets its own independent four attempts. Wrap five call sites and your fleet's retry ceiling is 5×4 with nothing coordinating them — exactly the "local caps compose into a global disaster" failure. The decorator is so ergonomic that it hides the fact that there's no shared state anywhere.

The fix is to make the budget an object the retry predicate consults, not a constant baked into the decorator:

class RetryBudget:
    def __init__(self, capacity, refill_per_sec, bonus=0.1):
        self.tokens, self.cap = capacity, capacity
        self.refill, self.bonus = refill_per_sec, bonus
        self.t = time.monotonic()
        self.lock = threading.Lock()

    def _refill(self):
        now = time.monotonic()
        self.tokens = min(self.cap, self.tokens + (now - self.t) * self.refill)
        self.t = now

    def allow_retry(self):
        with self.lock:
            self._refill()
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            return False

    def on_success(self):
        with self.lock:
            self._refill()
            self.tokens = min(self.cap, self.tokens + self.bonus)

BUDGET = RetryBudget(capacity=40, refill_per_sec=1.0)

def call_model(payload):
    for attempt in range(5):
        try:
            r = _do(payload); BUDGET.on_success(); return r
        except TransientError:
            if attempt == 4 or not BUDGET.allow_retry():
                raise
            time.sleep(backoff(attempt))

The trap Python makes easy: forgetting the Lock. Under threads or a thread-pool executor the read-modify-write on self.tokens races, and a "budget of 40" quietly lets through many more retries under exactly the load spike you built it for. Under asyncio the lock becomes an asyncio.Lock and the sleep an await asyncio.sleep, but the shape is identical: the budget is one object, shared, guarded.

Go: the context is your budget's expiry, not its size

Go doesn't tempt you with a magic decorator; it tempts you the opposite way, by making the retry loop so explicit that people reinvent it per package with subtly different backoff. The idiom that actually holds is to pass the shared budget alongside the context.Context and let the context own deadline, the budget own count:

func callModel(ctx context.Context, b *RetryBudget, p Payload) (Resp, error) {
    var last error
    for attempt := 0; attempt < 5; attempt++ {
        r, err := do(ctx, p)
        if err == nil {
            b.OnSuccess()
            return r, nil
        }
        last = err
        if !isTransient(err) || attempt == 4 || !b.AllowRetry() {
            return Resp{}, last
        }
        select {
        case <-time.After(backoff(attempt)):
        case <-ctx.Done():
            return Resp{}, ctx.Err() // deadline/cancel beats the retry
        }
    }
    return Resp{}, last
}

RetryBudget here is guarded by a sync.Mutex exactly as in Python. The Go-specific trap is the select: if you time.Sleep(backoff) instead of racing the sleep against ctx.Done(), a cancelled or deadline-exceeded request keeps sleeping and retrying against a caller who already walked away. In a fleet that's how you get workers burning budget on requests no one is waiting for anymore. The context is not the budget — the context bounds how long, the budget bounds how many — and you need both or you've built half a limiter.

JavaScript: the retry that outlives the request

In JS the danger is neither a hidden cap nor a reinvented loop — it's that promises float free of the thing that started them. A for await retry loop is easy:

async function callModel(payload, budget, signal) {
  let last;
  for (let attempt = 0; attempt < 5; attempt++) {
    try {
      const r = await doCall(payload, signal);
      budget.onSuccess();
      return r;
    } catch (err) {
      last = err;
      if (!isTransient(err) || attempt === 4 || !budget.allowRetry()) throw last;
      if (signal?.aborted) throw new DOMException("aborted", "AbortError");
      await sleep(backoff(attempt), signal); // sleep must reject on abort
    }
  }
  throw last;
}

Single-threaded event-loop means you get the budget's read-modify-write atomicity for free — no lock. That's the one place JS is easier. But it hands you a subtler leak: the AbortSignal. If your sleep doesn't reject when the signal aborts, and your doCall doesn't forward the signal, then a client that navigated away (or a request whose timeout already fired) leaves a retry loop running in the background, spending budget on a result that will be thrown into the void. The event loop won't complain — it'll happily run your orphaned retries next to the live ones. Wiring signal through every await is the JS equivalent of Go's ctx.Done() and it's the thing code reviews miss.

The pattern under the three

Strip the syntax and it's one design in three costumes:

The budget object is the same in all three. What changes is the runtime's model of cancellation — threads and locks in Python, contexts in Go, signals in JS — and in each language the retry leak comes from a call that keeps trying after the caller has given up. Get the budget shared and the cancellation wired, and the retry math from the budgets post holds regardless of what you wrote it in. Miss either, and you've got a limiter that leaks under precisely the load it exists to survive.

And because these calls aren't idempotent by default, none of this is safe until the retried operation carries a dedup key — a retry budget bounds how much you spend recovering, not how many times you accidentally send the same email.

Backoff constants, bucket sizes, and transient-error classification are workload-specific; the numbers above are placeholders. The invariant — one shared budget, cancellation wired through every wait — is what transfers across languages and providers.