In March we began moving our Consul servers into a newer hall of our data centre, one at a time. The plan was to add a new server, let it catch up, and only then remove an old one, so we would never drop below three. On Tuesday afternoon we added the first new server. That evening a switch between the two halls rebooted during planned maintenance, and service discovery for the whole platform stopped accepting changes for thirty four minutes.
Consul, like etcd and ZooKeeper, accepts a write only when a majority of its servers agree. With three servers the majority is two, so you can lose one. With four it is three, so you can still lose only one. The fourth server bought no extra tolerance and added a new way to fail, because four can split two and two. That is what the switch did: two old servers in the old hall, one old and one new in the new hall. Neither side had three, so neither could elect a leader.
Clients allowing stale reads kept working, so running services still found each other. Anything that needed a write failed. Deploys could not register new instances. Health check results were not recorded, so a node that went bad during the partition kept receiving traffic. Two teams whose batch jobs chose a leader through Consul sessions had no leader at all.
What stung was that the plan looked cautious. Never fewer than three sounds like safety. The number that mattered was not how many servers we had, but how many could fail, and where, before the rest stopped being a majority. With three we could lose any one machine. With four split evenly across a link we did not own, we could lose one switch.
We finished the move by adding and removing servers in pairs, so every intermediate state had an odd count, and we wrote down which failure each state survived before starting it. The cluster now runs five servers across three halls, placed so that no single hall holds a majority, and a script checks placement against the rack inventory before any membership change is allowed.
A majority vote does not reward you for adding voters. It rewards you for where they sit, and an even count usually means paying for a server that gives an outage one more way in.
– Sergey Shinder
