On the morning of July 10, 2026, the design-generation feature of promptflow — our multi-LLM workflow SaaS — went completely dark. The cause wasn't our code. Google had discontinued the gemini-2.5 model family the same day, with no advance notice. And this "the model just vanished" incident wasn't the first. On July 5 of the same month, the image model dall-e-3 disappeared from OpenAI's API, and one month before that, on June 3, we had just finished scrambling to handle the retirement of gemini-2.0-flash-lite in curation, our news-curation platform.
This article puts these back-to-back "provider retires a model" incidents from two separate products side by side and records, from the logs and code at the time, (1) why they become an outage instantly, and (2) what to monitor and what to prepare as a fallback to contain the damage. We hadn't anticipated this risk well enough. Model retirement hit three times in one month. The first, in curation (6/3), was handled by simply swapping in the successor model; but in promptflow it struck twice in July (7/5 and 7/10), and that is where we stopped "fixing it when it breaks" and built the defense into the system instead. The individual incidents are written up separately — the promptflow edition (PF-002) and the curation edition (MISC-002). This article focuses on the cross-cutting lessons from putting the two together.
Here is the conclusion up front (about a 6-minute read).
- The cause wasn't our code — it was the provider retiring a model. A model that worked yesterday simply doesn't exist today. It's the kind of change code review can never catch.
- Hard-coding model IDs into code and prompts is what spread the outage across everything. The moment a version disappears, the whole generation path goes down with it. Pin versions to aliases, and hold declared fallback candidates for when one is retired.
- The response format changed too, and parsing caused collateral damage. Gemini 3 started returning its "thinking" part separately, and the generated HTML lost its doctype. Don't let parsing implicitly depend on one model's output format.
- The same incident happened three times in one month, across two products. Rather than leaving it as a patch, we promoted it into model-catalog governance and automatic switch-over on retirement.
The symptoms: "does not exist," and a trap where the whole response format changes
The first sign was on the curation side. On June 3, gemini-2.0-flash-lite, which we used for article analysis, entered its retirement-notice window and responses became unstable. That one was solved just by moving to the successor gemini-2.5-flash-lite; while we were at it, we merged the two analyze and long_insight calls into one and cut roughly 16% of the tokens per article. At that point it felt like nothing more than a routine "rename the model" chore.
The real trouble came the next month, on the promptflow side. On July 5, the image-generation E2E test failed like this:
POST /v1/images/generations model=gpt-image-1 -> HTTP 504 Gateway Timeout
POST /v1/images/generations model=dall-e-3
-> 400 { "error": { "code": "model_not_found",
"message": "The model `dall-e-3` does not exist" } }
Our primary, gpt-image-1, returned a 504, and the fallback candidate dall-e-3 was reported as "does not exist." We restored service on the spot by switching to gpt-image-1-mini. But five days later, on July 10, Google discontinued the gemini-2.5 family without notice and design generation went completely dark. The nasty part was that it wasn't just a name disappearing. The migration target, the Gemini 3 family, had changed its response format itself — it misread the "thinking" part as body text, and the generated HTML lost its <!doctype html> entirely, a second-order failure.
The first hypothesis, and how it was wrong
On July 10, the first thing I suspected was that our own prompt engineering had broken. Design generation had been in constant quality tuning around that time, so I thought, "did I break the prompt again?" But rolling the prompt back to the last stable version still produced empty output. Next I suspected the API key had expired — but other models responded fine.
The split became clear the moment I pinned the model ID and threw a bare call at it:
curl -s "$GEN_ENDPOINT/models/gemini-2.5-flash:generateContent" \
-H "x-goog-api-key: <REDACTED>" \
-d '{"contents":[{"parts":[{"text":"ping"}]}]}' | jq '.error // .candidates[0]'
What came back wasn't our bug — it was an error from the provider meaning "this model is no longer offered." That is where the hypothesis shifts from prompt to provider. In other words, what worked yesterday simply doesn't exist today. It's the kind of change code review can never catch.
The mechanism: why model retirement becomes a total outage so fast
Once I stepped back and organized it, there were three reasons the damage spread across everything at once.
- We had hard-coded model IDs into code and prompt contracts. When a generation path is pinned to a specific version, the moment that version disappears the whole path goes down.
- The response parsing implicitly depended on one model's output format. Once Gemini 3 returned the thinking part in a separate field, the old parser mixed it into the body and the HTML broke.
- We had no way to know about retirements in advance. Provider announcements are buried in email and changelogs; you cannot assume a human tracks them every day.
In short, against a model disappearing for reasons outside our control, the system had no room at all to switch to an alternative.
The fix: stop pinning versions, make parsing tolerate broken responses, and detect retirements
As an immediate response, we moved model references from fixed IDs to latest-equivalent aliases, and added thinking-part exclusion and HTML salvage to the response parser.
// Gemini 3 mixes in a thinking part. Extract only the body, and
// if the doctype is missing, add it to rescue the "broken HTML."
function extractHtml(parts: Part[]): string {
const body = parts
.filter((p) => p.kind !== "thought") // don't misread the thinking part as body
.map((p) => p.text ?? "")
.join("");
const html = body.trim();
if (!/^<!doctype/i.test(html) && /<html[\s>]/i.test(html)) {
return `<!doctype html>\n${html}`; // add the doctype if it's missing
}
return html;
}
On top of that, we turned recurrence prevention into a mechanism. One part is model-catalog governance: hold the usable models as a versioned set, run an automatic new-model scanner on a cron (scheduled job), and surface additions and retirements to a human via notifications. The other is graceful fallback on retirement: declare alternative candidates per model, and when a call fails with model_not_found, switch automatically to the next candidate. The picture is a map like this:
# Declare each generation model's version and where to switch on retirement
image_generation:
primary: gpt-image-1
fallback: [gpt-image-1-mini] # drop the retired dall-e-3 from the candidates
design_generation:
primary: gemini-flash-latest # don't pin the version
fallback: [gemini-flash-stable, sonnet-html-fill]
parser: defensive-v2 # thinking exclusion + doctype salvage
On the curation side too, we stopped treating a model migration as a mere "rename" and paired it with logic that drops unexpected responses into a terminal status to prevent infinite retries. The lesson from curation fed straight into the design of promptflow.
Afterward, and the limits
After this July double-hit, model retirement went from "an incident you fix when it happens" to "an operation that assumes it will happen and switches to an alternative." The scanner cron now tells us about new models ahead of time, and at least we have reduced how often "everything suddenly stops one morning" happens. But the limits are clear too. The fallback's quality isn't the same as the primary's; for a feature like design generation, where output quality is the selling point, a "doesn't go down but degrades" state remains. The scanner can't detect a response-format change — in the end you don't know until you throw a real request at it. And fundamentally, as long as you depend on a model you don't own, you cannot reduce the risk of supply cutoff to zero. All you can do is prepare so that when the day comes, it is a partial degradation rather than a total stop.
Takeaways you can reuse
- Don't hard-code model IDs. Pin versions to aliases, and hold declared fallback candidates to switch to on retirement. "It worked yesterday" doesn't guarantee "it exists today."
- Don't let response parsing implicitly depend on one model's format. Write it assuming you'll salvage the body even from a broken response — for format changes like a mixed-in thinking part or a missing doctype.
- Don't track retirements by hand. Manage the usable models as a versioned set, and detect additions and retirements with a scanner cron that surfaces them to a human.
- Turn the first occurrence into a mechanism. The same incident will inevitably recur in another product. Don't leave it as a patch; promote it into fallback and governance, and you'll greatly reduce the damage the second time around.
