Our second line has five engineers and by March they were a fortnight behind on anything that was not an outage. Before asking for another head I read a month of escalations, four hundred and twelve of them, over three evenings.
About two thirds of them did not need an engineer. They needed a permission or a paragraph.
A hundred and forty eight were one of four requests that first line is not allowed to perform: adding somebody to a distribution group, releasing a quarantined message, resetting a second factor enrolment, restoring a deleted file. Ninety six were a known fault with a known workaround that existed in the heads of two people and nowhere else. Forty more had been escalated because the category list in our ticket tool has not described this estate since 2019, and the dropdown a caller chooses decides which queue they land in. That left about a hundred and thirty that genuinely wanted somebody senior.
Neither cause was an accident. The permissions were withdrawn after an audit in 2021 recommended tightening administrative roles, and the tightening was done with the roles the product ships with rather than the tasks we actually delegate, because splitting them properly was three weeks of work nobody had. The knowledge articles were a project that ended in 2022 when the colleague writing them changed job.
We delegated seven narrow permissions, each logged and each reviewable, which leaves the audit position better than before. Any fault escalated three times now gets an article before the ticket is closed, written by the engineer who fixed it and then followed by a first line colleague without help, because an article nobody has tested is a draft. The category tree was replaced with a list of the systems we run. And first contact resolution is reported per issue type instead of as one number, since the one number had sat at about a third for four years and told us nothing about which third.
First line now closes around three fifths of what arrives. A second factor reset takes six minutes instead of forty. I did not get the extra engineer and I no longer want one this year.
Escalation had looked like a measure of how hard our problems were. Read one at a time, it was an inventory of what we had never permitted and never written down.
– Serguey Shinder
