The green checkmark problem
Every MSP dashboard has one: the backup column, green for months. Green feels like safety. But a green job proves exactly one thing — the job ran. It does not prove the data comes back, how long that takes, whether the increment you'll need survived the retention policy, or whether anyone alive can find the decryption key at 2 AM.
Two weeks ago this series argued that an incident response plan that's never been exercised is a hypothesis. Backups are the same species of hypothesis, with quieter failure modes:
- Rotated credentials. The backup service account changed; jobs kept "succeeding" against a subset of targets.
- Corrupt increments. The chain is only as good as its weakest link, and nothing verifies links like a restore.
- Retention surprises. The copy you need is from 9 days ago; retention keeps 7.
- Key-inside-the-vault. The decryption key or console password is documented… inside a system that's now encrypted.
- Scope drift. The new file share, the new VM, the migrated mailbox — added to production, never added to the job.
Every one of these is invisible on a green dashboard, and every one of them is found in one of two moments: a scheduled drill on a Tuesday afternoon, or the worst hour of your year.
The drill
THE 45-MINUTE RESTORE TEST (roles: an operator + a scribe. That's the team.)
0:00 Pick the target BEFORE you feel ready: one production system that
would hurt Monday morning — file server, finance share, the PSA/RMM
database. Not the easy one. The one that matters.
0:05 Restore last night's copy to an ISOLATED target: a spare VM, an empty
VLAN, a cloud sandbox. NEVER over production. No internet egress
required — isolation is the point.
0:35 Verify like a USER, not an admin: open three files, run one report,
log in with a real non-admin account. "It boots" is not "it works."
0:40 Write four numbers: minutes to restore · GB restored · items verified
· today's date. The scribe signs the page.
That signed page is the deliverable. Everything else — the war stories, the "huh, interesting" moments, the ticket you'll open about the service account — is bonus.
The one rule
Borrowed from the tabletop drill and repointed: every claim gets a number and a date.
"Restores work fine" is a mood. "47 minutes, 212 GB, three files opened and readable, August 11, signed by the scribe" is evidence — the kind cyber-insurance applications and enterprise security questionnaires are explicitly asking for when they say tested recovery and documented RTO. You're not doing extra work to satisfy the questionnaire; the drill's natural output is the answer.
The math you can finally do
Recovery time estimates usually divide a known number by a guess:
Real downtime ≈ (data that must come back) ÷ (restore throughput)
Every team knows the first number. Almost nobody has measured the second — and the difference between an assumed and a measured throughput is the difference between promising "a few hours" and delivering a long weekend. After one timed drill, your RTO conversations use a measurement. Guessing is over.
The trap: your backups are the first target
Ransomware crews have told reporters for years that backup infrastructure is their first stop — delete or encrypt the recovery path, then detonate. Recent KEV traffic makes the architectural point for them: Veeam's advisory for CVE-2026-44963 describes a critical RCE on domain-joined backup servers, reachable by any authenticated domain user. One stolen domain account, and the vault is another room on the sinking ship.
So alongside the restore drill, three architecture questions worth an hour:
- Credential separation. Does the backup console share credentials (or auth domain) with production? It shouldn't.
- An undeletable copy. Does at least one copy live where a domain admin cannot delete it — immutable object storage or genuinely offline?
- Console-loss drill. If the console died tonight, who could still reach a copy, and by what path?
The commonly used rule of thumb is 3-2-1-1-0: three copies, two media, one offsite, one offline or immutable, zero errors on verification. The last digit is the one everyone skips — and it's the only one that involves actually restoring. Treat it as a starting point to adapt, not a certification to claim.
When the test fails
A restore test that fails is a successful test — it located a real gap for the price of 45 minutes. Run the tabletop rule on it: the gap gets an owner and a date, and the failed log page stays in the file next to the eventual passing one. The paper trail from red to green is itself evidence of a functioning program. The only failed test is the one never run.
Make it a rhythm
One drill is a data point; a rhythm is a program. Rotate targets quarterly — file server this quarter, the PSA database next, a full VM the one after. Fifteen minutes of scheduling covers a year: four calendar invites, each with an operator, a scribe, and a target named in advance. Booked beats intended.
