At four minutes past nine on a Wednesday an engineer applied a proxy configuration to what he believed was a pilot group of twelve machines. By thirteen minutes past, four hundred and six laptops had it. None of them could reach an internal application for the rest of the morning, and the service desk took five hundred and seventy calls before lunch.
The change itself was correct and had been tested properly. The target group was the problem. It had been edited three weeks earlier, by someone else, for an unrelated licensing exercise, and now resolved to every managed device in the company. Nothing in the console showed how many machines a change would reach before you pressed apply, so the number four hundred and six was first seen by us on a graph afterwards.
The part that turned a nuisance into a morning was the recovery. The tool that had delivered the change was also the only way to withdraw it, and it reached those laptops through the path the change had broken. Ninety or so machines had to be brought to a desk and corrected by hand, which for our remote staff meant a courier and two days.
Four things changed. Changes now go out in stages, a technical group first, then five percent, then a quarter, then everything, with a minimum soak of a day between them and a stated reason if anyone wants to skip a stage. The console shows an affected device count before execution and anything over fifty requires a second person to read the target group aloud with you. The management channel and the recovery path are excluded from every policy by standing rule, so the tool can always reach the estate it has just damaged. And group membership is now change controlled in its own right, with an owner, because the group was the actual defect and no process had ever treated a group as a configuration item.
The hours our endpoint tooling saves us every week are real and I would not give them back. The risk it carries is not what any one action does. It is how many machines it does it to, how quickly, and whether there is any route left to reach them afterwards.
Speed is the whole benefit and the whole exposure. Only the second one arrives without warning.
– Serguey Shinder
