Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

ChatGPT Down? 8 Fallback Patterns for API Outages [2026]

At some point, someone on your team is going to post “ChatGPT down?” in Slack.

Your users will never see that message. They’ll just see your product timing out.

In the next 45 minutes, you’ll have a production-ready chatgpt down api outage fallback playbook: detection (status page plus your own telemetry), retry policies that don’t melt your servers, circuit breakers, bulkheads, throttling, multi-provider routing, hedged requests for tail latency, and a graceful-degradation ladder (cached partials, no-tools mode, offline queue).

One detail people gloss over. OpenAI’s status page explicitly warns that its uptime is aggregate and that “individual customer availability may vary depending on their subscription tier as well as the specific model and API features in use” (OpenAI Status). Translation: the dashboard can be green while your app is on fire.

I’m writing this with actual scar tissue. Running this site’s multi-agent publishing pipeline (261+ posts shipped) taught me a boring lesson: deterministic reliability gates catch more real outages than “just use a bigger model” ever will. When an upstream dependency starts timing out, the only thing that matters is whether you fail fast and degrade cleanly.

What is a chatgpt down API outage fallback?

A ChatGPT down API outage fallback is the set of client-side and system-level resilience patterns that keep an LLM feature usable when OpenAI (or any LLM provider) is degraded or unavailable, by combining timeouts, retries with backoff/jitter, circuit breakers, throttling, bulkheads, caching, and multi-provider routing.

This isn’t just for chatbots. It’s for anything agentic: background document processing, code review bots, AI agents, and agentic AI workflows that chain tools and retrieval.

Check provider status (status.openai.com) and validate from your own telemetry

When someone asks “Is the ChatGPT API down right now?”, the worst answer is “looks fine to me.” The only acceptable answer is: give me 30 seconds, I’m looking at our dashboards.

Start with the obvious external signal:

• OpenAI reports separate uptime for APIs (99.94%) and ChatGPT (99.68%) on the current status window (OpenAI Status).
• The history feed shows the usual suspects: elevated errors, latency/timeouts, invalid_prompt spikes, and “interrupted streaming” incidents (OpenAI Status — History).

Then validate with your telemetry, because OpenAI’s disclaimer basically tells you to.

The three signals I’ve found most actionable

1. Error rate by class (429 vs 5xx vs network timeout). Track as percent of requests per minute. Alert when > 2% for 5 minutes on your primary model.
2. Latency by phase for streaming:
   • TTFT (time to first token)
   • stream duration
   • interrupted streams (count)

If TTFT p95 jumps from 1.5s → 6s, your “availability” might still be 99.9%, but UX is toast.

1. Token burn rate. During retries, cost spikes fast. On this blog’s agent pipeline, the simplest budget guard I use is a per-job “max attempts” cap plus a hard time budget. It stops one flaky dependency from quietly doubling spend.

If you want a concrete metrics schema for agent workflows, I’d pair this with my AI in production metrics guide and an agent orchestration setup that emits request-level spans.

Automatic detection: health checks that don’t lie

Do synthetic checks against the exact model + feature combo you depend on. OpenAI incidents are often model-specific (the history feed regularly calls this out).

A practical setup:

• Run a synthetic every 30 seconds per region.
• Use a cheap prompt and cap output to 32 tokens.
• Record: HTTP status, TTFT, total latency.
• Trip “degraded” if 3 of the last 5 checks exceed your p95 SLO.

That “exact model + exact feature” point matters even more for systems using RAG or tool calls, because those introduce extra ways to fail mid-flight.

Retry pattern: timeouts, max attempts, idempotency, backoff/jitter

Retries are where well-meaning teams accidentally DoS themselves.

The retry guidance from Microsoft is boring, and it’s correct: retries must be selective, capped, and time-budgeted (Microsoft Azure Architecture Center). OpenAI’s own guidance also focuses on 429 rate limits and recommends backoff rather than hammering the API (OpenAI Cookbook).

Timeouts: streaming vs non-streaming

If you set one global timeout, you’ll end up doing one of two dumb things:

• kill streams that are “working but slow,” or
• let hung streams pin your worker threads until your own service falls over.

A simple budget I like:

• Non-streaming completions: 10–20s hard timeout (depending on max output). If a user is waiting in a UI, 20s is already pushing it.
• Streaming: separate budgets
  • connect + TTFT timeout: 3–5s
  • overall stream timeout: 60–120s
  • idle timeout (no bytes): 10s

These aren’t universal numbers. Use them as starting points. The real rule is phase-specific timeouts, not one blunt hammer.

How many retries during an outage?

During a real outage, more retries make it worse. The question “How many retries should I use for an API outage?” is mostly answered by: what’s your time budget.

A production-friendly default:

• Retry only on:
  • 429 (rate limit)
  • 503 / 502 (temporary upstream)
  • network timeouts
• Do not retry on:
  • 400-class prompt validation errors (like invalid_prompt spikes you’ll see in the history feed)
  • auth errors
• Cap to 2 retries (so 3 total attempts)
• Total time budget: ≤ 25s (interactive)

Exponential backoff with jitter (to avoid retry storms)

Exponential backoff alone can accidentally synchronize clients into a thundering herd. The fix is jitter. As Marc Brooker explains, jitter randomizes retry timing so you don’t create a coordinated retry storm under contention.

A good shape:

• base delay: 250ms
• multiplier: 2x
• max delay: 5s
• jitter: full jitter (random 0..delay)

A concrete error-classification matrix (retry vs failover vs degrade)

“ChatGPT down” isn’t one thing. It’s a handful of failure modes that need different moves.

Circuit breaker (closed/open/half-open) to prevent cascading failures

Circuit breakers are non-negotiable for LLM apps. Without them, when an upstream gets slow, you get hung requests. Then exhausted worker pools. Then your own app starts returning 500s and everybody pretends they’re surprised.

Microsoft’s definition is crisp: “Temporarily block access to a remote service or resource after failures reach a threshold” (Microsoft Azure Architecture Center). The warning right under it is the real reason you’re here: timeouts block concurrent requests, consuming threads, memory, and connections until you fail.

Configure thresholds for LLM calls (failure rate + slow calls)

LLM incidents often look like “it still responds, just slowly.” That’s why you need slow-call rate as a first-class breaker input.

Starting config I’d ship:

• sliding window: 50 requests or 30 seconds
• open circuit if:
  • failure rate ≥ 25% or
  • slow-call rate ≥ 40% where “slow” = p95 SLO breach (example: > 8s non-streaming or TTFT > 4s streaming)
• open duration: 30 seconds
• half-open trial: 5 requests

When half-open succeeds, you close. When it fails, you reopen. The state machine is simple. Your alerts and dashboards are the hard part.

One subtle rule: circuit breakers should be per-model

A lot of teams wire one breaker per provider. That’s wrong.

Outages are frequently isolated to a single model or capability (again, check the history feed). Break per (provider, model, capability). For example: openai:gpt-4.1-mini:streaming.

Bulkheads/isolation: separate pools/quotas per tenant and workload

Bulkheads are how you stop one customer or one workload from taking down everything.

A clean mental model:

• interactive chat traffic (SLO-focused)
• background extraction/summarization (throughput-focused)
• internal jobs (admin/backfills)
• premium vs free tiers

Each gets:

• its own concurrency pool
• its own token budget
• its own circuit breaker thresholds

If you’re building production AI features for enterprise tenants, bulkheads are the difference between “one noisy customer” and “global incident.”

I’ve seen a tiny version of this even on smaller systems. In my blog pipeline, “distribution” jobs (posting, indexing) are isolated from the heavy “research” tool loops. Otherwise one slow upstream API turns into a backlog that delays publishing.

Throttling and load shedding to protect your SLOs under load

When upstream is degraded, your app tends to take a double hit:

1. request latency increases, which reduces throughput
2. users retry manually (refresh, spam submit), increasing load

You need throttling and load shedding. Microsoft’s throttling guidance maps cleanly to LLM systems: control resource consumption to maintain SLOs under load (Microsoft Azure Architecture Center).

Practical controls (ship at least 5):

1. Per-tenant concurrency cap (e.g. free tier max 2 in-flight requests)
2. Global concurrency cap per provider+model (prevents a stampede)
3. Token budget per minute per tenant (hard cut-off)
4. Queue background jobs with a max backlog (e.g. 10k tasks). After that: reject/park.
5. SLO-based shedding: if TTFT p95 > 4s, disable expensive tool chains.

Here’s the part people avoid saying out loud: load shedding is a product decision. You’re picking which experiences survive the incident.

If you want a place to anchor the cost side of this, connect it to my LLM cost writeup. Throttling isn’t just “reliability.” It’s how you keep a retry storm from turning into a surprise bill.

Multi-provider routing/failover (primary/secondary models)

Failover is not “swap the API key.” Your prompts, tool schemas, and safety policies have to be compatible across providers, or you’ll fail over into a different kind of outage.

A workable architecture:

• Provider abstraction layer: normalized request/response types, streaming adapter, error mapping
• Routing policy: primary → secondary based on health + cost + latency
• Capability flags: tool calling, JSON mode, max context, multimodal

If you’re already doing agent framework work, treat the provider as another dependency. It gets the same hardening.

Primary/secondary routing policy

I keep it simple:

• Primary provider/model for normal mode
• Secondary for:
  • circuit open
  • sustained elevated errors (> 2 minutes)
  • “slow-call storm” (slow-call rate threshold hit)

If you’re routing across multiple LLM APIs, track success rate + p95 separately and use a weighted policy.

Retries vs failover (the difference that trips teams)

• Retries assume the same dependency recovers inside your time budget. They’re for transient faults.
• Failover assumes the dependency is unhealthy longer than you can tolerate. It’s a topology change.

If you keep retrying during a provider incident, you’re just adding load to a sinking ship.

Reduce user impact: hedged requests, cached partials, and fallback modes

This is the part most “ChatGPT down” posts skip. They jump from “retry” to “we’re offline.” That’s lazy engineering.

Hedged requests to cut tail latency (without doubling cost)

Hedging means: if a request is taking too long, start a second request to a different provider/model, then cancel the loser.

Cost control rules that actually work:

• hedge only after p95 latency threshold (example: if not started streaming by 3s)
• cap hedges per tenant (e.g. max 1 hedge / 10 requests)
• cancel loser immediately on first token from winner
• never hedge when you’re already rate-limited (429)

If you do this right, you’re paying extra only for the tail. That’s where UX dies.

Safe fallback modes (and when to use each)

When the LLM is unavailable, you need a degradation ladder. Here’s mine:

1. Same provider, smaller model (cheaper/faster) when you’re slow but not erroring
2. Secondary provider when the circuit is open or 5xx spikes
3. No-tools mode when tool calls are failing (often your own infra, not the model)
4. Cached partials when a multi-step workflow fails mid-way
5. Offline queue for background tasks (email the result later)

The “cached partials” move is underused. Google’s API error guidance explicitly calls out partial errors as a legitimate design approach (Google Cloud). For LLM apps, that usually means returning:

• retrieved passages (from your retriever)
• tool outputs that already ran
• a partial streamed answer with a clear “generation interrupted” marker

Users will tolerate partial progress. They won’t tolerate silent failure.

Can I cache LLM responses safely?

Yes, but cache the right things.

What I cache in LLM systems far more often than full text completions:

• embeddings for identical documents
• retrieval results (top-k doc IDs) for common queries
• tool outputs for expensive deterministic calls
• prompt scaffolds (system prompts, policies)
• safety checks (moderation classifications), with short TTL

Full completions are trickier. They’re sensitive to tiny prompt changes, and if you key poorly you can leak tenant data. That’s not theoretical.

Cache rules that prevent foot-guns:

• include tenant ID in the cache key
• include model ID + temperature in the key
• use TTLs (start with 5–30 minutes for retrieval/tool outputs)
• prefer “stale-but-safe” content for read-only UX (docs, suggestions)

If you’re doing retrieval work, connect this with retrieval-augmented generation and the security side of prompt injection. Caching poisoned retrieval results is a real problem.

Incident runbook for LLM outages (alerts, comms, flags, postmortem)

The difference between “we had an incident” and “we had a meltdown” is whether you had a runbook.

Here’s a template I’d actually use.

Detection and triage checklist

• Check external status: OpenAI Status and history.
• Confirm in internal dashboards:
  • error rate by class
  • TTFT p95 / stream interruptions
  • queue depth
  • token burn rate
• Decide incident severity in 5 minutes.

Immediate mitigations (feature flags)

Have these flags pre-wired:

• disable tool calls (no-tools mode)
• route to secondary provider
• drop max tokens (reduce latency)
• switch to smaller model
• disable streaming (if stream interruptions are the issue)
• enable “offline queue” for background tasks

If you don’t have feature flags, you don’t have reliability. You have hope.

Comms templates

Internal:

• “Provider degraded: elevated 5xx and TTFT. Circuit open for primary model. Routing 80% to secondary. Next update in 15 minutes.”

External status page:

• “AI features may be delayed. We’re routing to backup providers and will deliver results with partial output when needed.”

Post-incident checklist (the part teams skip)

• classify errors (429 vs 5xx vs timeouts vs stream interruptions)
• quantify user impact: % requests degraded, max backlog, time to recovery
• quantify cost impact: extra retries + hedges (dollars)
• decide if routing policy thresholds were right
• add one chaos test for the incident shape

If you’re serious about this, fold it into your AI security and production testing posture. Outages are when attackers also get creative.

How to test failure modes (fault injection, chaos, replay)

If you only test the happy path, you’re not “production-ready.” You’re just lucky.

What I’d test in CI and staging:

• inject 429s for 5 minutes. verify backoff+jitter and token burn caps
• inject 503s. verify circuit opens and routing flips
• inject stream interruptions mid-response. verify partial output behavior
• inject 10s latency. verify slow-call circuit breaker opens
• replay real traffic samples through the router (shadow mode)

For a deeper testing framework for agent systems, I’d use my AI engineering evals and agent evaluation harness patterns. Reliability is just another eval axis.

If you’re building anything users depend on, treat “ChatGPT down” like a routine dependency failure, not a once-a-year apocalypse.

My prediction: within 12 months, customers will start asking for LLM dependency SLOs the same way they ask for SOC2. If you can’t explain your fallback ladder and routing policy on a single page, you’ll lose deals to the team that can.

Originally published on kunalganglani.com