The plan was three days of work. Take the twelve servers behind one application, write the configuration management for them, and stop patching and configuring them by hand. Three weeks later I stopped the effort and wrote down why, because the reason was more useful than finishing would have been.
The automation worked immediately on seven of them. The other five each failed differently. One was two minor operating system versions behind because it had been excluded from a maintenance window in 2023 and never caught up. One had a filesystem mounted at a different path, because the original disk had been undersized and somebody solved it at three in the morning. One had a firewall rule added by hand for a monitoring tool, present nowhere else and documented nowhere at all. One had a configuration file edited directly to raise a timeout during an incident, a change that was never made anywhere else and that turned out to be the only reason that node was stable. And one was simply built by a different person in a different year.
Every one of those decisions was correct at the time it was made. Collectively they meant there was no such thing as one of these servers, only twelve individuals that shared a name. I was not writing automation. I was writing a catalogue of undocumented history, in conditionals, and the catalogue would have been longer than the thing it automated and would have rotted at the same rate.
So we did the boring thing first. We defined what the server should be, in one file, and rebuilt the estate to match it rather than encoding the drift. Where a difference was genuinely required, it went into an exception register with an owner and a reason, and there are now three exceptions instead of five accidents. Drift is detected daily and reported, and the only supported way to change one of these machines is through the definition.
Automation does not create consistency. It assumes it, and it is honest about how much you have. If your scripts need to know which machine they are running on, standardisation is the work in front of you, and automation is what you get afterwards.
– Serguey Shinder
