Our change success rate last year was ninety-eight point six percent, and I have quoted that figure in three governance meetings. It took an afternoon of reading to work out that it does not mean what everybody in those rooms assumed.
A change is marked successful in our process when the implementer records that it completed and no incident is raised in the hour afterwards. That is a statement about whether anything broke. It says nothing whatsoever about whether the change achieved the thing it was raised to achieve, and when we took a sample of two hundred standard changes and asked what evidence existed for that, roughly half had none beyond the service continuing to run.
The examples were not dramatic, which is the point. A firewall rule added to the correct device and the wrong group, so it matched nothing and blocked nothing. A backup schedule amended on a policy that no longer had any systems attached to it. A monitoring threshold updated on the template, which applied to new hosts and left the existing three hundred as they were. A security patch installed correctly on a server where nobody restarted the service, so the process kept the old library loaded for four months. Every one of those was implemented by a competent person, closed the same day, and counted towards the figure I was quoting.
The cause is structural rather than careless. The person who raises a change describes an outcome, the person who implements it performs a task, and only the first of them can tell whether the outcome occurred. Our form had a field for implementation notes and nowhere at all to say what would be observably different afterwards.
So the change record now carries a verification statement, written by the requester before approval, in the form of something that can be looked at. Evidence is attached before closure. A change that is implemented but not confirmed goes into a state of its own rather than counting as either outcome, and one in ten is checked by somebody who was not involved.
The rate fell to ninety-one percent immediately, and the number is now worth something. We had spent years measuring whether the work happened, and no time at all measuring whether it worked.
– Serguey Shinder
