This is part of a series on building Orca, a single-binary orchestrator for the gap between Coolify and Kubernetes.
Here's the incident, cold:
A worker node in my cluster had a control-plane WebSocket session to the master that had been dead since the 7th. It was the 17th. For ten days, every deploy targeting that node timed out with agent did not acknowledge within 10s. And the entire time, the node showed up green in the dashboard. Its heartbeats were flowing. orca status cheerfully reported a service as 1/1 running - a container that had, in fact, been crashed for fourteen hours.
Nothing was on fire. Nothing was alerting. The system was lying to me with a completely straight face, and it had been for a week and a half.
Why the node looked alive
The master↔agent link is a WebSocket. The agent sends a heartbeat every 5 seconds; the master pushes a status ping every 30. The master tracks connected agents in a map keyed by node id. Simple.
The bug is that liveness was inferred from two different things that aren't the same thing. Heartbeats proved the agent process was alive. But deploys were dispatched over the WebSocket session. And those are only the same as long as the socket is actually open.
Then came a half-open TCP connection. The agent's box dropped off the network in a way that sent no FIN, no RST - a NAT conntrack timeout, a VM freeze, the kind of unclean disappearance that's depressingly common. On the master's side, the read on that socket just... blocked. Forever. No error. The session stayed in the connected map. Every status ping the master sent went into a kernel send buffer and evaporated.
Meanwhile the agent, on its side, was in the exact same state: its read blocked forever, and its 5-second heartbeat writes succeeded into a dead send buffer that would only surface an error after the TCP retransmit timeout - about fifteen minutes later, if ever.
So both ends believed they were connected. The map said "connected." And there was a rule - added to fix an earlier bug - that a node with a live session must never be pruned, no matter how stale its heartbeat. That rule was correct in spirit and catastrophic in practice, because "has an entry in the map" was being treated as proof of life. A zombie session is immortal under that rule.
The three prior "fixes" that missed
This was the fourth time this class of bug had bitten. Each previous fix had patched a symptom:
- One stopped a node from being wrongly pruned while connected.
- One improved the timeout error message.
- One handled stale service state.
Every one of them treated a consequence. None of them touched the disease, which was this: node liveness was tracked separately from the channel the master actually dispatches over. As long as those were two different sources of truth, a half-open socket could always desync them.
The fix: liveness is the deploy channel
The permanent fix was to stop having two definitions of "alive." A session is alive if and only if it's carrying traffic. And here's the nice part - both sides already generate regular traffic (5s heartbeats, 30s pings), so I didn't need a new protocol at all. I just needed to enforce a deadline on it:
- Master: each session's read loop runs under a read-idle deadline (30s, configurable). Silence past the deadline means the socket is half-open. Tear the session down: remove it from the map, drop the node's placeholder service state, mark it unreachable.
- Agent: a 90-second read-idle deadline (three missed pings). A half-dead socket breaks the loop, and the existing backoff-reconnect takes over. No human, no restart.
- A missed deploy ACK now has consequences. It doesn't just error - it kills the session, on the theory that an agent that can't ACK a deploy over the channel isn't really connected. The next deploy fails fast with "unreachable until it rejoins" instead of re-timing-out against a corpse.
The stale 1/1 running fixed itself as a consequence: placeholder state only exists while a live session refreshes it, so when the session dies, the placeholders die with it. The system can no longer report confident, hours-old lies, because the state that would tell the lie is gone.
The test that would have caught it
The reason this bug survived three fixes is that nobody had written the test that actually reproduces it. Killing a process is easy to test. A half-open socket is the hard case - the peer is gone but the TCP connection is still open. So the regression test does exactly that: it connects, then goes completely silent while holding the socket open, and asserts the master tears the session down within the deadline.
If your liveness check can't survive that test, it isn't a liveness check. It's a process check wearing a liveness check's clothes.
The lesson
Prove the thing you actually depend on, not a proxy for it. Heartbeats proved the process was up. I was deploying over the socket. Those were different questions, and the gap between them was a ten-day outage that never alerted.
The follow-on lesson is about recurrence: when the same class of bug bites more than twice, stop patching symptoms and go find the shared assumption underneath. Mine was "an entry in the connected map means the node is reachable." Once that assumption was false-by-construction - because a silent session gets torn down - the entire family of bugs closed at once.
It's been running in production since, and it's already earned its keep: when I decommissioned a node, its services dropped cleanly out of status instead of lingering as ghosts. The machinery told the truth about a node leaving. After ten days of it lying about a node staying, that felt like a real win.
