We moved our billing service to blue green deployments in the spring. Two identical environments, the router points at one, you deploy to the idle one, test it, flip the router, and keep the old one running for an hour so that rollback is a single command. For requests it worked beautifully.
On a Thursday night we shipped a fix for a rounding error in late payment fees. The deploy finished at twenty past midnight and we flipped the router at twenty five to one. At one o'clock both colours ran the nightly overdue job, because the scheduler runs inside the service and both copies of the service were running. Three thousand two hundred customers got two overdue notices, one with the old fee and one with the corrected one.
That was the visible half. Both colours were also consuming from the payments queue in the same consumer group, so for that hour roughly half of all payment events were processed by the code containing the bug we had just released a fix for. As far as the router knew, the release was complete at twenty five to one. As far as the queue knew, it was half complete until twenty five to two.
Blue green switches where inbound requests go. It does nothing to the work a service starts on its own: schedulers, queue consumers, pollers, outbound syncs, cache warmers. For all of that, the idle colour is not idle.
The active colour is now a key in the same configuration store the router reads, and flipping the router and writing the key are one step. Every instance checks the key before it starts a scheduled job or a consumer, and the idle colour pauses both within ten seconds of a flip while still serving health checks, so rollback stays instant. Scheduled jobs also take a lease in the database, so a mistake in the colour logic produces a skipped run rather than a second one. And the release checklist lists every entry point a service has, not only its routes. Billing turned out to have seven.
We sent three thousand two hundred apologies, and the corrected fee stands.
A deployment strategy is a statement about which traffic reaches which code. Ours described the traffic we receive and said nothing about the work we start ourselves, which for a billing service is most of it.
– Sergey Shinder
