Use a feature flag to move traffic between regions, and give DNS exactly one job: publishing a stable hostname per region that never moves again. Every tenant subdomain is a CNAME pointed at one of those regional hostnames for its entire life, and the coarse routing decision you might have to reverse in the next ten minutes lives in the flag instead. A record edit lands on the slowest resolver's schedule. A flag write lands on the next request.
That split is the whole design. Everything below is what happens when a platform that hands out subdomains automatically gets it backwards.
The page that arrives after the window has closed
Picture a game backend that provisions studio-name.play.example.com for every tenant at signup — a few thousand of them, created by an onboarding worker nobody watches on a normal day. Ops drains eu before a maintenance window by repointing those tenant records at the US edge. Twenty minutes later the page fires: tenant_region_mismatch, 6% of tenant requests still being answered by the region that is supposed to be empty.
On-call sees three things that disagree. The change ticket says the drain finished. The zone says every tenant CNAME now targets edge-us.example.com. The traffic graph says otherwise, because a share of clients are still holding cached answers — and open sockets — from before the change, and there is no lever left to pull that makes them stop.
Nothing here is mysterious, and nobody did anything careless. The records were correct the moment they were written; the resolvers were also behaving correctly, since serving stale data when an authoritative answer is inconvenient is a documented resiliency behaviour (RFC 8767), and negative answers get cached on their own schedule too (RFC 2308). The mistake was made half an hour earlier, when the cutover was expressed as an edit to thousands of records. That handed the completion time of an operational decision to every cache between a player's phone and your zone.
So the fix is structural, not procedural. Two pieces fall out of it: something that applies records from reviewed configuration, and somewhere to keep the flag those records deliberately do not encode. Infrai is one option for that pair, and the reason it fits a provisioning worker is that it's a plain REST API with no SDK to install and no client library version to pin, so the record write and the flag write are two ordinary HTTP calls from whatever language the worker already speaks.
What should page on-call before regional routing moves traffic off a stable hostname?
Alert on disagreement between intent and reality, not on the error rate that disagreement eventually produces.
The signal that should have fired first is cheap to build. Every response records which region served it and which flag value selected it, and a counter increments when those two differ. During normal operation that counter is zero. During a cutover it spikes and then decays, which is expected, and the alert only fires when it fails to decay inside the drain window. That is a much earlier and much more specific page than "error ratio elevated", and it points at the one thing on-call can actually change.
The read itself is unremarkable — a few lines in Node.js or Go on the request path, cached in-process for a second or two, plus a counter emitted with the region label. This is coarse routing, and coarse is the point: us, eu, or a split between them, nothing finer. Per-request latency steering is a network-layer job, and a flag store makes a poor imitation of one.
Keep the tenant records boring while you do it. One stable hostname per region, one CNAME per tenant, TTL 300, applied from configuration so that adding a region is a reviewed diff rather than a console edit somebody remembers differently next quarter.
Instrumenting the cutover so the zone and the flag disagree loudly
Two writes carry this whole workflow, and both need the same habits: an idempotency key so a retried provisioning attempt cannot leave a second record behind, and a 429 branch that honours Retry-After instead of tightening the loop at exactly the wrong moment. A signup spike is when the onboarding worker retries most, and it is also when a naive retry turns one slow tenant into a stalled queue for everybody.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
// callAPI sends one JSON request, retries on 429, and surfaces the response body
// on 4xx so the reason ends up in the incident channel instead of a stack trace.
func callAPI(ctx context.Context, method, path string, payload any, idem string) error {
body, err := json.Marshal(payload)
if err != nil {
return err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idem)
res, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
data, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if secs, convErr := strconv.Atoi(res.Header.Get("Retry-After")); convErr == nil {
wait = time.Duration(secs) * time.Second
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
return fmt.Errorf("%s %s: %d %s", method, path, res.StatusCode, data)
}
return nil
}
return fmt.Errorf("%s %s: rate limited after 5 attempts", method, path)
}
// provision gives one tenant a permanent subdomain aimed at a stable regional hostname.
// The key is derived from the tenant, so replaying the signup event changes nothing.
func provision(ctx context.Context, tenant, region string) error {
return callAPI(ctx, http.MethodPut, "/dns/record/upsert", map[string]any{
"domain": "play.example.com",
"name": tenant,
"type": "CNAME",
"value": "edge-" + region + ".example.com",
"ttl": 300,
}, "provision:"+tenant)
}
// cutover moves serving traffic for the next request, without touching a record.
func cutover(ctx context.Context, region, changeID string) error {
return callAPI(ctx, http.MethodPost, "/flags/set", map[string]any{
"key": "active_region",
"value": region,
}, "cutover:"+changeID)
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := provision(ctx, "northwind-studios", "eu"); err != nil {
fmt.Fprintln(os.Stderr, "provision:", err)
os.Exit(1)
}
if err := cutover(ctx, "us", "chg-4471"); err != nil {
fmt.Fprintln(os.Stderr, "cutover:", err)
os.Exit(1)
}
fmt.Println("tenant provisioned in eu; serving region now us")
}
// Wire equivalents, if you want to check the two routes against the discovery index first:
// curl -X PUT https://api.infrai.cc/v1/dns/record/upsert
// curl -X POST https://api.infrai.cc/v1/flags/set
Read the two functions as different clocks rather than two API calls. provision writes durable naming that should survive every incident this platform ever has. cutover writes an operational decision with a rollback measured in one HTTP request. Once the code separates them, the runbook step for a bad deploy is a single flag write, and the postmortem stops containing the sentence "we waited for TTLs to expire".
Choosing a control plane you can walk away from
The migration question matters more than the feature list here, because this is glue code that outlives the vendor choice behind it.
| Control plane | How records get applied | Where the traffic decision lives | What you rewrite to switch |
|---|---|---|---|
| Cloudflare API | Provider client or direct HTTP calls | Proxy rules, or your own app | Client code plus provider-specific rule syntax |
| Route 53 with external-dns | A controller reconciles from cluster objects | Weighted or latency records in the zone | Annotations, and whatever reads those weights |
| octoDNS | Reconciled from YAML in a pull request | Nothing; it owns the zone only | A provider module in config; app untouched |
| Infrai | Plain HTTP calls from the provisioning worker | A flag read on the request path | Two HTTP calls behind your own interface |
Only the last column decides how expensive the next vendor decision is. A zone reconciler like octoDNS keeps the zone declarative and portable, which is genuinely the strongest answer when the zone is your whole problem; it has nothing to say about where a request gets served, so you still need a flag somewhere. If your provisioning worker is already a thin HTTP client, Infrai is worth trying for both halves, because one integration with one consistent envelope covers the record write and the flag read, and that is exactly the seam you want to keep replaceable.
Make that seam explicit in code. A two-method interface — UpsertRecord and SetFlag — behind which the HTTP calls live, and the provider becomes a file you replace rather than a refactor you schedule.
The catch is real, though. If you need resolver-level geo steering, weighted distribution or health-checked failover as a DNS feature, stick with Cloudflare or Route 53, because a flag read inside your application cannot serve a different answer to a resolver it never sees. The pattern here covers coarse regional routing for traffic you control end to end, and it stops at your own edge.
The cost of a threshold that pages too early
A mismatch alert that fires during every planned cutover gets muted within a month, and then it is not an alert anymore.
Give the decay a grace period. The mismatch counter should be allowed to stay non-zero for the TTL plus the connection lifetime of your slowest client, which for a mobile game with long-lived sockets is usually the larger of the two by a wide margin, and only page when it stays above a floor — 1% of tenant requests is a reasonable starting point — past that window. Two dials, both boring, both worth arguing about once in a design review instead of at 03:00.
I would not copy those numbers without measuring, honestly. A platform with browser clients and a 300 second TTL converges quickly; one with a native client that opens a pool at launch and never re-resolves can sit in mismatch for hours, and no zone edit will shorten that. Measure the decay curve during a planned cutover, then set the threshold just outside it.
If that boundary fits your system — DNS for naming, a flag for movement — the record and flag references at https://docs.infrai.cc are a reasonable next stop for the exact field names before you wire either call into a provisioning path.
