A junior engineer on my team once ran a cleanup command on staging, believing he was on dev. The data was gone before the prompt returned.
He came to me directly, shaking, honestly, and told me what happened. Which, I want to point out before anything else, is the single most important fact in this story.
On a lot of teams, that engineer spends an hour trying to quietly fix it himself first, and the incident gets worse in the dark. He came straight to me. That is not a personality trait. That is a property of the team, and you build it or you destroy it long before the incident happens.
The first hour
The first hour is triage, not analysis.
I took over communication with the affected dev team myself. Not because the engineer could not speak, but because the person with the least organizational armor should not be the face of an incident. We stabilized, restored from backup, verified, and closed the loop with the dev team the same day.
At no point did anyone ask who did it in a public channel. The dev team was told: we made a mistake, we have restored it, here is what we are changing.
Accountability flows up. A manager who forwards blame downward is just a router.
The interesting question
Once it was over, the question I sat with was not how do I make sure he is more careful.
Careful is not a system. Everyone is careful until they are tired, or it is 2 AM, or two terminal windows look identical, which is the actual root cause here. He was not careless. He was a human being using a tool that made two very different worlds look exactly the same.
So we changed the tools.
Terminal backgrounds now encode the environment. Dev is default. Staging is blue. Production is red, a red you cannot ignore, a red that makes you sit up slightly before you type. It costs nothing and it works on the tired version of you, which is the version that makes mistakes.
A pre execution checklist for destructive commands. Three questions, ten seconds. Where am I? What system is this? Am I on a read node or a write node? We wrote it into the runbook.
I told the team the analogy I actually think in. You cut vegetables outside the vessel, then put them in. You do not chop directly into the cooking pot, because there, every slip is dinner.
Neither fix is clever. That is the point. Clever fixes depend on people being sharp. Boring fixes work when they are not.
What happened to the engineer
Nothing. That is the answer, and it was deliberate and visible. No formal note, no development area in his next review, nothing.
The incident was the process's fault, the process got fixed, and the fix was named after the problem, not the person.
He stayed. He got better, genuinely better, the kind of better that comes from learning viscerally that mistakes are survivable here. Years later he was one of the people I trusted most in a production window.
If I had made that first incident expensive for him, I would have taught him, and everyone watching, because everyone is always watching, that the smart move next time is to hide it.
The general rule
When something breaks, there are always two available explanations. The person, or the system the person was inside.
Choosing the person is emotionally satisfying and fixes nothing. The next human inherits the same trap. Choosing the system is boring and permanent.
We never had that class of incident again. Not because the team became more careful, but because the trap was removed.
The terminal colors outlived everyone on that team, including me.
