Our 'staging' environment had real tenants via a forgotten DNS door, a two-year-old prod dump, and debug shortcuts β because environments are labeled by intention, not contents. The fix: classify by data, and make staging ephemeral with nightly rebirth on Krova Cloud.
One guard, courtesy of a commenter: the rebirth has a supply chain. The scrub and seed scripts are built against a schema, and migrations move the schema under them. If they live in a wiki or a separate repo, a migration silently breaks the nightly rebirth and nobody notices until the next demo. So they're versioned right next to the schema migrations β same repo, same PR β and the rebirth ends with a smoke test (healthz plus a sane row count) that alerts on failure. Cleanup became a property of the system; this keeps the system honest about its own breakage.
The alert said staging was unreachable. My first thought was who cares. The second, four seconds later: then why are customers tweeting that they can't log in?
The investigation found staging had been serving real users for months β a forgotten DNS record, a redirect from an old marketing page, customers onboarded during a demo and never moved. Staging hadn't "become production." It had become production for some people, which is the same thing with worse defaults.
What the audit found
- A prod dump, copied "just once" two years ago. Real names, real emails, real rows. A copy of production data is production-shaped, whatever the hostname says.
- Relaxed rules everyone knew about. Debug endpoints. An admin panel behind a path instead of a password. A test account whose password lives in an offsite whiteboard photo. In prod these are incidents; in staging they were convenience β right up until real people lived there.
- No backups, no alerts, no drills. Nobody backs up a costume. Then the costume had tenants.
The rule we now enforce in every planning meeting: production isn't a name. It's wherever your users' data lives.
Classify by data, not hostname
Anything holding real user data gets production care β backups, alerts, auth, patching β even if it's called staging, demo, or bob-box. Real data enters scrubbed or not at all; the "one quick dump" is banned.
And staging itself is now ephemeral by design: it dies and is reborn nightly, scrubbed on boot, deleted every Friday. An environment that dies weekly cannot accumulate secrets, drift, tenants, or folklore.
#!/bin/bash
# staging-reborn.sh β nightly
set -euo pipefail
krova cubes delete staging # old staging dies completely
krova cubes create staging \
--image ubuntu-24.04 \
--ssh-key "$(cat ~/.ssh/deploy.pub)" \
--vcpu 2 --ram 4 --disk 40 \
--user-data "$(cat scrub-cloud-init.yml)"
#cloud-config
# scrub-cloud-init.yml β staging is reborn clean every night
runcmd:
- curl -sf -H "Authorization: Bearer ${DUMP_TOKEN}"
https://storage.private/scrubbed-dump.sql.gz
| gunzip | sudo -u postgres psql app
- systemctl restart app
On Krova Cloud this lifecycle is affordable because it's billed by the minute, and a stopped Cube bills only its disk β Friday's delete is the default state, not an event.
The doors that make environments lie
Orphaned DNS is how staging acquires tenants. The weekly check:
while read -r d; do
t=$(dig +short "$d" CNAME | head -1)
# any record still pointing at staging is a door
[ -n "$t" ] && case "$t" in *staging*) echo "DOOR: $d -> $t";; esac
done < domains.txt
Plus the classification check β ask staging what it's holding, not what it's named:
krova ssh staging -- sudo -u postgres psql app -tAc \
"SELECT count(*) FROM users WHERE email NOT LIKE '%@example.com';"
# any number above zero means staging is lying to you
krova ssh staging -- ss -tlnp
# debug ports listening? that's a prod-shaped wound on a staging-shaped box
The honest part
- Synthetic data has a realism tax. Some bugs only bite real-shaped data β the 400-character name, the emoji in the address line. We use a smaller, weirder synthetic set plus occasional scrubbed real samples, reviewed like code. Perfection isn't on the menu; classification is.
- Small teams can't afford full parity. Fine. The rule isn't "staging must equal production." It's "nothing real lives in a place with pretend rules." One sentence, enforceable at any size.
- Ephemeral staging doesn't fix prod. Prod still needs real walls: no public IP by default, own kernel per Cube, scoped secrets. This post is about misclassification, not about replacing the castle.
Environments lie. Data doesn't.
Go ask your staging what it's holding β not the hostname, the contents. If the answer includes real names, real emails, or one customer who wanders in through a forgotten door, you don't have a staging environment. You have an unmonitored production wearing a costume.
Ours wore it for months. The customers never noticed. That's not a compliment. That's the scariest part.
