Zero downtime database migration: 5 flags and a 17x P90 gap

TL;DR

A zero downtime database migration is five feature-flagged phases — old-only, dual writes, shadow reads, new-primary, new-only — where every step is reversible by flipping a flag back. The trap is not the cutover; it is that your new store's latency profile can look fine at P95 and be 17x worse at P90, and that the comparison machinery you added to stay safe is itself production code that can take a region down. Budget the migration in months of soak time, not in days of engineering.

Vercel published how they migrated the database behind every Vercel build on 11 August 2026 — Redis to DynamoDB, February to April. It is a good writeup precisely because it includes the parts that went wrong, and those parts are the transferable ones.

What a zero downtime database migration actually is

A zero downtime database migration is one where both stores run in production simultaneously and traffic moves between them in reversible, independently-flagged steps. There is no maintenance window because there is no single moment of cutover — there is a sequence of moments, each of which can be undone in the time it takes a flag to propagate.

That definition rules a lot of things out. Dumping and restoring during a quiet hour is not this. Neither is a dual-write deploy that ships all five behaviours behind one boolean, because a single flag means a single rollback granularity: when something breaks you can only go all the way back, and you learn nothing about which step broke it.

The five phases each answer exactly one question, and they are not interchangeable:

Vercel ran exactly this ladder: "the rollout ran as feature-flagged phases: Redis-only, then dual writes, shadow reads, DynamoDB-primary, and finally DynamoDB-only." The value is in the separation. Dual writes prove the new store accepts your data; shadow reads prove it returns the same data. Those are different failures, and a dual write that silently drops a field looks perfectly healthy until a shadow read compares the two.

Why they migrated at all: durability, not speed

The reason matters, because it determines whether any of this work is justified. Redis was, in their words, "fast and made sense at the time" — operations were "about a millisecond each, and the code grew to lean on that speed." The move to DynamoDB was not a performance upgrade. It was the opposite: they knowingly traded latency for durability.

The dividing line they drew is the useful heuristic. Tokens and container statuses can be rebuilt if they get lost — "lose a token and the pool rebuilds it within about ten minutes." Billing mappings cannot. State that can be regenerated belongs in a cache and does not need a phased migration; state that cannot be regenerated needs a durable store, and that is what earns three months of ladder.

So before you build any of this, sort your keys into rebuildable and not-rebuildable. If everything is rebuildable, delete the old store and let the new one fill. A zero downtime database migration is machinery for protecting the second category, and running it for a pure cache is a quarter of engineering time spent on nothing. This is the same instinct as treating a research spike as running code: find the one question that decides the design before building the thing.

The percentile you pick decides whether you ship

Here is the number that should change how you run these migrations. On getWarmPoolTokenCount, the P95 measured 1.29ms on Redis and 5.13ms on DynamoDB. Roughly 4x — slower, but the kind of slower a team argues its way past in a review.

At P90, where the gap was widest, it was about 17x.

That inversion is not a typo and it is not rare. Redis and DynamoDB have differently-shaped distributions: Redis is tight and low with a tail that only appears at the very top, while DynamoDB's response times sit higher and flatter across the middle percentiles. Comparing them at one percentile tells you about that percentile and nothing else. Their own conclusion is the line worth stealing: "even a 1ms-to-15ms query time degradation on P90 could bring down our warm pool management logic."

Why P90 rather than P95 mattered so much comes down to what the code did with those queries. A tight supply loop calling a store many times per iteration lives at the percentile it actually hits most often, not at the tail you happen to alert on. Multiply a 14ms delta by the number of calls per iteration and a background loop that used to finish in milliseconds starts stretching into seconds.

Three practical rules follow:

1. Baseline per operation, not per store. "DynamoDB is a few milliseconds slower" is not a measurement. getWarmPoolTokenCount is.
2. Compare the full distribution. P50, P90, P95, P99, on the same operation, on the same traffic. The widest gap is the one that will page you, and it will not always be at the top.
3. Multiply by call count before you decide. A per-query delta is meaningless until you know how many queries sit inside the loop that consumes it.

The baseline phase — old-only, instrumented — exists entirely to make step 1 possible. Skip it and you have nothing to compare against later, which means every subsequent latency question becomes an argument instead of a lookup.

Your comparison machinery is production code

This is the part most migration guides leave out, and it is the most valuable thing in Vercel's writeup.

Shadow reads are a safety mechanism. They are also new code on a hot path, issuing extra queries against a store nobody has load-tested at full read volume, doing comparison work inline, and logging the results. All of that is production code with production failure modes.

In March, one region's builds went down. "The investigation pointed at our own comparison machinery." Four days later, a pull request citing the incident added a missing index.

Read that sequence carefully. The mechanism they added to avoid an incident caused one. That is not an argument against shadow reads — the comparisons were also what caught the real bugs, since "every mismatch got chased down to a real bug, a dual-write race, or an expected difference before we moved on," and, as they put it, the comparisons told them "whether the two stores agreed on the stored values, which is something tests alone couldn't." It is an argument for treating the comparison layer with the same seriousness as the migration itself:

• Give the comparison its own flag, separate from the dual-write flag. You want to be able to stop comparing without stopping the migration.
• Sample it. Comparing 1% of reads finds systematic disagreement just as fast as comparing 100%, at 1% of the added load. Full comparison is for the final soak, not for week one.
• Make it structurally unable to fail the request. The shadow path runs after the response is decided, its errors are swallowed and counted, and a timeout on the shadow read is a metric, not a 500. GitHub's Scientist library formalised exactly this shape years ago — run the candidate, never let it affect the control.
• Index the new store for the shadow query pattern, which is not always the primary query pattern. That is the specific bug that bit them.

There is a second, blunter lesson in their timeline. Later that same month, "the Redis infrastructure that we were migrating off actually went down." The store you are migrating away from does not politely stay healthy while you take three months to leave it. Mid-migration is the most fragile your system will ever be, because you now depend on both stores — which is an argument for shortening the ladder's calendar time, not for adding phases to it.

Common mistakes

Treating the migration as a code change. The code is a week. The schedule is set by soak time: each phase must run long enough to cover daily peaks, weekly batch jobs, and month-end spikes. A phase validated only on a Tuesday afternoon is not validated. Vercel's took February to April.

One flag for all five phases. Then rollback granularity is all-or-nothing and you cannot attribute a regression to a step. Five flags, five independent kill switches.

Assuming dual writes are atomic. They are not. Two stores, two calls, and a process that can die between them. Every mismatch class Vercel chased included "a dual-write race." Make writes idempotent, key them so a replay converges, and expect the comparison phase to find drift — that is what it is for.

Alerting on one percentile. Covered above, and it is the mistake with the highest blast radius, because it produces a confident "we measured it" that is wrong.

Deleting the old store the day the flag flips. New-primary and new-only are separate phases for a reason. Keep writing to the old store through the new-primary soak so that "serve old again" stays a live option rather than a restore procedure.

Skipping the baseline. Without per-operation numbers from before you started, every later latency question is unanswerable.

When a zero downtime database migration is the wrong tool

Do not run five phases for a pure cache — delete and refill. Do not run it when the two stores can be reconciled offline and the write volume is low enough for a lock-and-copy inside a real maintenance window; a 30-second window you can actually take is cheaper than three months of dual-write bookkeeping.

And do not run it to move between two stores when the actual problem is the access pattern. If a loop is calling the datastore hundreds of times per iteration, that loop is the thing to fix, and fixing it may remove the reason for the migration entirely. Vercel found their supply loop stalling in April and discovered "the loop had stalled under Redis too, sometimes for a minute or more" — the migration surfaced a pre-existing bug rather than causing it. Infrastructure changes are a slow, expensive way to discover application bugs, and the cheap fix often wins outright — halving a Node process's memory turned out to be one V8 flag, not a new runtime. The same trade-off logic applies to picking the destination in the first place, including an orchestration layer you have to operate yourself — and, as with Docker Swarm versus Kubernetes, the honest comparison is usually a boring number rather than an architecture diagram.

Conclusion

The ladder works. Five phases, five flags, reversible at every step, and Vercel came out the other side with Redis gone from the warm pool paths by April. But the two things that actually threatened them were not the cutover: a latency gap that looked like 4x at the percentile they checked and 17x at the one that mattered, and a safety mechanism that took down a region on its own.

So run the ladder, but instrument the operation rather than the store, compare the whole distribution before you commit, and ship your comparison layer as if it were the feature — because in production, it is.

Next: treat the risky part as running code before you commit to it.

Frequently asked questions

What is a zero downtime database migration?

It is a migration where the old store and the new store both run in production at the same time, and traffic moves between them one reversible step at a time instead of during a maintenance window. The standard ladder is five feature-flagged phases: old-only, dual writes, shadow reads, new-primary, and new-only. Nothing about it is instant — Vercel's migration of the datastore behind every build ran from February to April 2026 — but at no point is there a cutover that cannot be undone by flipping a flag back.

What is the difference between dual writes and shadow reads?

Dual writes send every mutation to both stores while the old one remains the source of truth, which proves the new store can accept your write shape and volume. Shadow reads query both stores on the read path and compare the results without serving the new one, which proves the two stores actually agree on the values. They answer different questions and you need both, because a dual write that silently fails produces a store that accepts everything and returns the wrong thing.

Why compare P90 latency instead of P95 when migrating a datastore?

Because the percentile you pick can flip the verdict. On Vercel's warm-pool token count, P95 measured 1.29ms on Redis against 5.13ms on DynamoDB — roughly 4x, which sounds survivable. At P90, where the gap was widest, it was about 17x. Compare the full distribution across percentiles rather than a single headline number, because a loop that runs many queries per iteration amplifies whichever percentile it actually lives at.

Can shadow reads take down production?

Yes, and this is the failure mode teams underestimate. Shadow reads double read volume against a store nobody has load-tested, add comparison work to a hot path, and run as ordinary production code with ordinary bugs. In March 2026 one Vercel region's builds went down and the investigation pointed at their own comparison machinery; a pull request citing the incident added a missing index four days later. Ship the comparison layer behind its own flag, sample it rather than running it on every request, and make sure a failure inside it can never fail the real request.

How long should a zero downtime database migration take?

Longer than the code change suggests, because the schedule is set by how long you leave each phase running rather than by how fast you can write the adapter. Vercel's ran roughly three months, February to April 2026. Each phase needs to sit in production long enough to cover your traffic's full cycle — daily peaks, weekly batch jobs, month-end spikes — since a phase that only ever ran on a Tuesday afternoon has not been validated.

When should you not do a phased migration?

When the data is cheap to rebuild. The phased ladder exists to protect state that cannot be regenerated if it is lost — billing mappings, ledgers, anything auditable. Vercel's own framing was that tokens and container statuses could be rebuilt if they got lost, but the billing mappings could not. For a pure cache, deleting the old store and letting the new one fill is faster, safer, and needs none of this machinery.

Sources

• Vercel Engineering — How we migrated the database behind every Vercel build, published 11 August 2026 (the five feature-flagged phases, the 1.29ms/5.13ms P95 figures on getWarmPoolTokenCount, the ~17x P90 gap, the March comparison-machinery incident and the index that followed, and the February–April timeline)
• GitHub — Scientist (the control/candidate pattern behind shadow reads: run the candidate, compare, and never let it affect what is served)

Originally published at umesh-malik.com

Keep reading on umesh-malik.com:

• Kubernetes on bare metal: the 4 cloud integrations you must build
• Fix EEVDF latency regressions with sched_ext: Meta's 28% p99 win
• Docker Swarm vs Kubernetes: $166/yr Beats a $200K Cluster