A recent interview put me in front of this exact use case, and I gave an average answer that left a bad taste in my mouth. It kept nagging me, so I built it five times and measured each one.

"Add a circuit breaker" usually means adding a library to every service that calls the flaky dependency. That works when the caller is one process. It breaks down when the caller is a fleet.

The setup

One producer, one durable RabbitMQ queue, and a fleet of consumers competing for its messages. Each message becomes one call to a third party whose load balancer, health checks and hosts are entirely theirs.

The textbook breaker assumes one caller and one dependency. We have neither:

• Many callers, one opinion needed. A breaker per consumer means each forms its own view from its own sample.
• Their load balancer, not ours. We see one address, so we can only decide when to stop calling them, not which host to avoid.
• Whose failure is it, anyway? A request shed by our own egress proxy can look identical to an upstream failure. With an upstream that was only 300 ms slower, every 5xx our callers saw came from our own proxy: 90,753 local errors in two minutes.

A breaker with no control plane floods

Step 1 has no breaker: every message spends its three delivery attempts against a dead upstream, and about 4,000 messages are dead-lettered in 20 seconds.

Step 2 puts a standard breaker (cockatiel) in every consumer. It does its one job: each replica stops after five failures. But nobody shares the verdict:

• one 15-second outage opens the fleet's five breakers about 27 times;
• all five agree on the state only 56% of the time;
• each half-open probe runs on its own clock;
• and 1,577–2,246 messages are dead-lettered from just 52–59 real failures.

That last number is the lesson. Most of the loss came from what "open" did, not from the outage. Every call an open breaker turned away still spent one of the message's delivery attempts, so messages the third party never saw ended up in the dead-letter queue.

What's missing is a control plane: something the fleet shares that says whether to call and who probes.

RabbitMQ already is one

Steps 3 and 4 add no new infrastructure. Everything the fleet needs to agree on is a broker feature:

Plus the thing a separate control plane would not give you: the work itself. While the breaker is open, messages wait durably in the queue that already holds them, instead of being tried, refused and dead-lettered by every replica.

The breaker got simpler, not bigger: by step 4 it keeps no state of its own.

The version matters. On a broker older than 4.3 every requeue counts toward the delivery limit, and a message waiting out an open breaker gets dead-lettered untried.

When the verdict has to leave the fleet

For most consumer fleets, step 3 or 4 is where to stop. Step 5 is for when other systems need to act on the verdict: a producer that stops accepting work, a status page, billing.

There, Envoy enforces per replica (outlier detection, concurrency limits) and a leader-elected aggregator takes a quorum of what the replicas report. It publishes one verdict per API as events with a gapless sequence. The fleet still coordinates through RabbitMQ: a topic exchange carries each verdict to a queue per consumer, and single-active-consumer queues elect the one consumer that probes, redrives or keeps a minimum of work going.

Across ten chaos faults (killing consumers, the leading aggregator, the broker, a hanging upstream, a lease partition mid-outage) it lost 0 messages and dead-lettered 0. It costs about three times the code of step 3 and 19 containers, and it pays only when something outside the fleet consumes the verdict.

Takeaways

• A breaker is four decisions: what counts as a failure, where the state lives, what "open" does, and who decides to try again. A library answers all four together; pulling them apart lets you answer each one better.
• For a fleet, "open" should mean stop consuming, not reject the message. Your broker already knows how to hold work.
• Look at your broker before adding a coordination service. Single-active-consumer, delivery limits, TTLs and dead-lettering already give you a probe permit, leader election and a durable timer.
• Measure what "open" costs, not just whether the breaker trips.

The full article, with each step's design and measurements, is at lsfera.github.io/circuit-breaker-unboxing. The code is TypeScript on Effect 4 and amqplib, one branch per step: