Our human resources platform is a subscription service and we have exactly one instance of it, the live one. The supplier offers a sandbox for thirty percent of the annual fee, which we declined at procurement in 2020 because nobody could say what we would use it for.
We found out in May. We had built an automation that reads starters, movers and leavers from that platform and creates accounts, mailboxes, building access requests and equipment orders. A change to handle internal transfers was tested the only way available, against real records, on a Thursday evening. The logic mishandled people who changed department and manager on the same date, and treated forty one of them as leavers. Their accounts were disabled at six the next morning, before most of them had arrived at work.
We restored them within ninety minutes. The more uncomfortable finding was how we had been testing everything else. The payroll interface, the expenses integration, the fleet management feed and the scripts that maintain the warehouse system's user list all run against services with a single environment. Testing meant a carefully chosen live record, a colleague willing to be a guinea pig, or a change run late on a Friday by somebody watching closely. It had worked for years mainly because the changes had been small.
Buying software as a service moved the test environment out of our hands and onto the supplier's price list. We treated it as an optional extra because the procurement comparison, reasonably enough, lined up licence costs like for like, and a sandbox made every bid look dearer.
Three things are different now. Every automation that writes to an external platform has a dry run mode that reports what it would do, and a change goes live only after a dry run against current data has been read and signed off by somebody from the owning department. We bought the sandbox at renewal for the two platforms where a mistake reaches people's pay or their access. And our procurement template now asks, for any system we expect to automate against, what non production environment the supplier offers and what it costs, before the licence comparison is done.
The place our automation gets tested is something we buy. For years we had been declining to buy it one contract at a time, without noticing that each refusal was a decision about how we would test.
– Serguey Shinder
