Our disaster recovery test last year was a success by every metric we had agreed to. The failover ran inside the recovery time objective, data loss was within the agreed window, applications came up in the documented order, and we produced a report with a lot of green in it that went to the audit committee.
The following month somebody from operations asked a question that undid most of my satisfaction with it. If that had been real, what would the warehouse have done for the four hours while the systems were coming back?
I did not know, and neither did anybody in IT, because we had tested our half and quietly assumed the other half was somebody's problem. It was not, in the sense that nobody owned it. The continuity documents for that department had been written in 2016 and described a manual fallback that involved a phone system we retired in 2019 and paper pick lists printed from a printer that was not in the recovery scope and would not have been running. Nobody had defined who tells five hundred staff what is happening or where to work. And nothing anywhere addressed the part that turned out to be the largest piece of work in every scenario we later modelled, which is what happens to the transactions that occurred during the gap and who reconciles them once the systems return.
That last one is the honest measure of an outage. The systems are back in four hours. The organisation is back when the backlog is cleared, the manual records are entered, and the discrepancies are resolved, which in our case would have been closer to three days.
We run a joint exercise now, and IT is not in charge of it. Each critical business process has a documented degraded mode that assumes no systems at all, owned and tested by the department that performs it. Recovery targets are stated as time to resume the process, not time to restore the service. And every scenario includes a reconciliation plan with a named owner.
IT can be accountable for recovering systems. Only the business can be accountable for resuming operations. Presenting the first as if it were the second is a comfortable habit, and it survives right up until a real event.
– Serguey Shinder
