A deploy at eleven on a Tuesday night broke search for about forty minutes. It had skipped the integration suite and the staging soak, and it had needed no second approver, all of which was allowed, because it had gone out through the pipeline we built for emergencies.
I went to look at how often that happened, expecting to find a handful. Thirty-one of the previous ninety production deploys had used it. A third of everything we shipped was going out through a path with no tests, no soak and no review, and not one of those thirty-one had been an emergency in any sense I would defend.
Nobody was being reckless. The standard pipeline took fifty-two minutes. Thirty-four of those were an integration suite that needed a retry roughly one run in five, so the honest expectation was an hour and a quarter, sometimes two. The emergency path took six minutes. If you are shipping a copy change at five in the afternoon, or fixing something a customer is waiting on, the fast path is not laziness. It is the rational choice, and we had made it available with no friction and no record.
That last part is what let it grow. Nothing anywhere counted how often the escape hatch was used. There was no report, no label on the deploy, no weekly number. A control that nobody measures does not exist as a control; it exists as an option.
Three changes, in the order they mattered. We labelled every deploy with the path it took and put the split in a weekly message to the team, which by itself dropped usage in the first fortnight before we had done anything else. The emergency path now requires a typed reason, announces itself in the channel, and runs the full suite afterwards against what was deployed, with an automatic rollback if it fails. And we spent two weeks on the actual problem: parallelising the suite, quarantining six tests that accounted for nearly all the flakes, and getting the standard pipeline to fourteen minutes.
Emergency usage is now about two in ninety, and those two were emergencies.
If your safe path is slow enough, the unsafe one stops being an exception. People are not working around your process. They are responding to it exactly as it is built.
– Sergey Shinder
