Our order capture service began failing intermittently at twelve minutes past eight on a Tuesday. It was declared a major incident at twenty past ten.
Everything our reporting shows about that event starts at twenty past ten. The published duration was forty minutes, which is true of the major incident and not remotely true of the outage. By the time the bridge opened we had lost most of a morning's orders and a hundred and ninety calls had reached the service desk.
Nobody behaved badly in those two hours. The duty manager was in his second week, and the threshold in our procedure is significant business impact, which is a judgement, and judgements made by new people default to caution. The application team genuinely believed they were ten minutes from a fix, and they believed it continuously from half past eight onwards, which is how ten minutes becomes two hours. And declaring a major incident here summons a director, a customer communications process, a bridge that twenty people join, and a report I read personally. Everyone involved knew that. Declaring was expensive and waiting was free.
So we made declaring cheap and waiting impossible. There are objective triggers now, and any of them declares automatically without anybody exercising judgement: a service on the critical list unavailable or degraded for twenty minutes, reports from more than three sites, or a regulatory or customer deadline at risk that day. Anyone in the company can declare, first line included, and nobody is questioned afterwards for a declaration that proved unnecessary. Standing down takes one line and produces no report.
The measurement changed with it. Our clock starts at the first report rather than at the declaration, because the second of those measures our paperwork.
Declarations went from nine last year to thirty one this year, which reads like a deterioration and is the opposite. Median time from first report to an open bridge fell from seventy one minutes to fourteen. Four of the thirty one were stood down inside half an hour, and I regard those four as the strongest evidence we have that it is working.
An escalation route that costs something to use will always be used late.
– Serguey Shinder
