We test our technical failover regularly and we are good at it. What we had never tested was the part that happens before anybody touches a keyboard, which is the sequence of phone calls that turns an alert at an antisocial hour into a person who is awake, informed and allowed to act.
So we ran the call, unannounced, at ten past two on a Wednesday. It took fifty-one minutes to assemble the four people the runbook said we needed.
Every delay was ordinary. The second-line number in the escalation document belonged to an engineer who had moved to another team in 2023 and had politely stopped being on call. The storage vendor's contract says twenty-four hours a day, which is true, and the severity that unlocks a response out of hours is assessed by their service desk against their own definitions, so our first call produced a ticket rather than an engineer. The bridge we use for major incidents required a licence that two of the people on the list did not have. The duty manager's phone was on silent, correctly, because his escalation depended on a paging system that had been decommissioned with the tool it came with. And nobody on our side was confident they were permitted to wake a director, which meant everyone waited for somebody else to decide.
None of this appears in a technical test, because the estate behaved perfectly throughout.
The contact register now has a single owner and is verified quarterly by calling every number on it rather than reading it. Supplier escalation is rehearsed once a year, in their words, so we know which phrase produces an engineer and which produces a ticket. Every role has two named deputies. The authority to wake senior people is written down, by name, so that nobody spends twenty minutes being considerate. There is a printed card, because the directory is exactly what we may not have.
We measure one number now: from alert to a person who can make a decision. It was fifty-one minutes. It is eleven.
An estate that recovers in minutes is still down for an hour if nobody has picked up the phone.
– Serguey Shinder
