A colleague opened our staging admin tool to check something unrelated and found a real customer's order in it, placed forty minutes earlier, with a real card reference on it. For nineteen days about a third of our checkout writes had gone into the staging database.
Three weeks before, an engineer had needed to reproduce a bug against staging data. He copied the checkout manifest, renamed the Deployment to checkout-debug, pointed it at the staging config map, and applied it in the production namespace. There was a ticket. Nobody was hiding anything. What none of us thought about was that he had copied the pod template labels along with everything else, and our production Service selects on app equals checkout.
A Service is a label query. The endpoints controller lists every ready pod that matches it and does not care which Deployment created them, what the pods are called, or what anyone intended. Six production pods and three debug pods answered the same query, so the debug pods received their share of live traffic and served it perfectly, against the wrong database.
Every signal we had was clean. The debug pods emitted metrics under the same service label, so the dashboards averaged them in. They returned 200s, because staging accepted the writes. Latency was if anything slightly better. The only evidence was in a database we do not monitor, and the only reason we found it was that somebody happened to look.
Three things changed. Every Service selector now includes an instance label that is unique per Deployment, so a copy cannot answer the query by accident. A policy rule rejects any pod whose config label does not match the environment label on its namespace, which would have refused this manifest at apply time. And a nightly report lists each Service together with the workloads behind its endpoints and flags any Service backed by more than one, which found nothing else, though I would not have bet on that beforehand.
Experiments now go in their own namespace with their own Services. Not because production is sacred, but because labels are the only thing the cluster reads, and labels do not know that you meant this one to be temporary.
– Sergey Shinder
