Put a Free Model Route Behind an Error Budget, Not a Retry Storm

429 is a signal, not a bug. A route that responds with 429 Too Many Requests is already telling the client to back off. A wrapper that immediately retries treats that signal as noise and spends remaining capacity faster. With the current wave of agent-driven workflows, one user action can fan out into many model calls, so a free model route is usually the first layer to saturate.

I used MonkeyCode's free model access and free server option as a disposable integration seam for this test. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I do not need a specific model name or quota to make the point: the problem shows up at any HTTP route that enforces rate limits.

The failure mode

Most integration wrappers start with a simple loop: call the endpoint, and if the status is 429 or 5xx, sleep and retry. That loop has three failure modes.

The worst case is a shared free route with many callers. A single process that retries on 429 can double or triple the original request volume before a human looks at the dashboard. If the caller is an agent loop, the same pattern can repeat for each tool invocation.

A small error-budget breaker

The breaker below reacts to three inputs: status code, Retry-After, and failure rate over a rolling window. It opens when either a 429 includes Retry-After, or the window contains enough samples and the failure rate crosses the configured limit. A half-open state sends one probe to test recovery.

from enum import Enum
import time

class State(Enum):
    CLOSED = 'closed'
    OPEN = 'open'
    HALF_OPEN = 'half_open'

class ErrorBudgetBreaker:
    def __init__(self, window_seconds=60.0, failure_rate_limit=0.20,
                 cooldown_seconds=30.0, min_samples=10):
        self.window_seconds = window_seconds
        self.failure_rate_limit = failure_rate_limit
        self.cooldown_seconds = cooldown_seconds
        self.min_samples = min_samples
        self.state = State.CLOSED
        self.window_start = time.monotonic()
        self.total = 0
        self.failures = 0
        self.opened_at = 0.0

    def _is_failure(self, status):
        return status is None or status == 429 or status >= 500

    def allow(self):
        if self.state == State.OPEN:
            if time.monotonic() - self.opened_at >= self.cooldown_seconds:
                self.state = State.HALF_OPEN
                return True
            return False
        return True

    def record(self, status, retry_after=None):
        now = time.monotonic()
        if now - self.window_start >= self.window_seconds:
            self.window_start = now
            self.total = 0
            self.failures = 0

        self.total += 1
        failed = self._is_failure(status)
        if failed:
            self.failures += 1

        if status == 429 and retry_after:
            self.state = State.OPEN
            self.opened_at = now - self.cooldown_seconds + retry_after
            return self.state

        if self.state == State.HALF_OPEN:
            if failed:
                self.state = State.OPEN
                self.opened_at = now
            else:
                self.state = State.CLOSED
                self.failures = 0
            return self.state

        if self.total >= self.min_samples and self.failures / self.total >= self.failure_rate_limit:
            self.state = State.OPEN
            self.opened_at = now

        return self.state

A production wrapper calls allow() before sending a request and record() after receiving a response.

def parse_retry_after(value):
    try:
        return float(value)
    except (TypeError, ValueError):
        return None

def call_with_budget(client, url, payload, breaker, timeout=10.0):
    if not breaker.allow():
        return {'error': 'circuit_open', 'state': breaker.state.value}

    try:
        resp = client.post(url, json=payload, timeout=timeout)
    except Exception as exc:
        breaker.record(None)
        raise exc

    breaker.record(resp.status_code, parse_retry_after(resp.headers.get('Retry-After')))

    if resp.status_code == 429 or resp.status_code >= 500:
        return {'error': 'provider_unavailable', 'status': resp.status_code, 'state': breaker.state.value}

    resp.raise_for_status()
    return resp.json()

Deterministic test plan

Rather than trusting a synthetic benchmark, I run the breaker against a deterministic sequence of statuses and check the state transitions. The table below uses the same code and parameters.

This is the minimum signal check I would want before wiring a new model route into a larger service. It verifies that the breaker sees rate limits differently from transient 5xx errors and that it does not stay open forever.

Where to place it

The breaker works best as a route-level seam shared by all callers to the same provider endpoint. If multiple workers use separate in-memory breakers, a noisy worker can still produce a retry storm while a quiet worker is open. In a single-process service, an in-memory instance is enough. In a multi-instance deployment, move the state to a small store or use coordinated half-open probes.

I keep the breaker outside the model call itself. The model code remains a simple HTTP call; the breaker decides whether that call is allowed to happen. That separation means the same wrapper can protect a free model route, a free server slot, or any OpenAI-compatible endpoint without changing the rest of the request path.

What this does and does not fix

An error-budget breaker prevents retry storms and makes rate-limit pressure visible. It does not change the provider limit, repair a misconfigured key, or stop an over-committed quota from being exhausted by other callers. It also does not replace logs, budget ledgers, or golden-set evaluation; those are separate seams for separate failure modes.

A 429 can arrive before the window has enough samples. The code opens immediately only when a Retry-After header is present. If the provider sends 429 without Retry-After, the call only contributes to the rolling failure rate. For routes that omit that header, I usually add a static cooldown or count a bare 429 as an immediate open signal; the decision depends on the provider contract, not on the wrapper.

This approach is not useful for one-off scripts with a single request, offline batch jobs where retry delay is cheap, or callers that cannot tolerate a half-open probe. It is most useful when the route is shared, the call volume is high, and the cost of a failed call is larger than the cost of waiting.

Run this harness against your own rate limit and print the state transitions before promoting a free route into production. The cheapest way to learn a provider's failure contract is with a breaker in front of it, not after a retry storm has already consumed the quota.