The recovery plan was documented. The restore procedures were current. The tests passed.
Then the one engineer who understood the undocumented dependency chain wasn't available, and none of that mattered.
Why Documentation Creates False Confidence
The instinct after a near-miss is always the same: write it down. Most teams already have — a runbook, reviewed on schedule, sitting next to the DR plan it supports. But a document and a capability are not the same thing.
Documented ≠ practiced.
Documented ≠ understood.
Documented ≠ repeatable.
A runbook captures the steps someone remembered to write down. It doesn't capture the judgment calls made in real time — which alert to ignore, which dependency to check first, which "probably fine" system state is actually a warning sign specific to your environment. The same gap shows up on the testing side, not just the documentation side: restore testing that only validates the procedure passes cleanly even when only one person on the team could actually run the real thing, because that person is usually the one who wrote the test.
The Hidden Recovery Dependency
Every recovery process has a layer that never makes it into the documentation, because nobody thinks to document something they've never seen fail. A few places it hides:
- Restore ordering — the sequence that works, versus the sequence in the runbook, are not always the same sequence
- Exception handling — which steps are safe to skip under time pressure, and which ones silently corrupt state if you do
- Vendor escalation — the contact who actually picks up at 3 a.m. usually isn't in the runbook, it's in someone's phone
- Legacy tooling — the one recovery tool nobody's touched in two years, that one person still remembers how to drive
- Automation overrides — knowing when the automated recovery path is wrong for this specific failure and needs a manual detour
None of these show up in a DR audit. All of them show up in an incident.
Bus Factor Changes Under Incident Conditions
Bus factor — how many people could disappear before something stalls — isn't fixed. It changes depending on what you're measuring.
Under normal operations, your recovery bus factor might genuinely be five: five people on the team who could, in principle, run the process. Under actual incident conditions, it might be one — because only one of those five has ever actually run it for real, under time pressure, with the runbook's exceptions in play. The other four have read the doc. They haven't lived it.
That gap between the two numbers is invisible until the person who is the one gets sick, quits, or is just asleep in another timezone when it matters.
A Better Recovery Question
Most teams ask: do we have a recovery plan? That question gets a "yes" almost every time, because almost every team does have one, on paper.
The better question is: could someone else execute this recovery successfully tomorrow, with no help from the person who usually does it?
That question gets a much more honest answer, and it's the one that actually predicts what happens during a real incident.
If you're interested in the architectural side of recovery readiness, I published a deeper analysis here: Your Recovery Architecture Has A Bus Factor Problem.
Originally published at rack2cloud.com
