Your WebSocket ping is answered by the edge, not your Durable Object

We run SSH traffic through a Cloudflare Durable Object. Each user's machine holds one outbound control WebSocket to the DO; when a phone wants in, the DO tells the machine to dial back. This is a standard hub pattern built on the WebSocket Hibernation API.

It worked, then it started failing, and every signal we had said the system was healthy.

The symptom

Phone connects to /connect/:machineId. The socket opens, then immediately closes with 1013. Meanwhile the machine that is supposedly offline believes it is connected:

• TCP is ESTABLISHED
• The WebSocket's protocol-level ping (pingInterval = 15s) succeeds, every time
• The app has no error to report

Only restarting the machine's app fixed it.

There is exactly one place in our Worker that emits 1013:

ws.close(1013, "machine agent offline");

That line runs when getWebSockets("agent") returns nothing for that machine. So the DO did not have the socket.

Why

Under Hibernation, the Cloudflare runtime answers protocol-level ping/pong on your behalf, without waking the Durable Object.

This is documented, intentional behavior. The entire point of Hibernation is to stop billing GB-seconds for an idle object, and waking it up to say "yes, still here" would defeat that.

But it means this state is reachable:

The machine decided it was connected based on row 2. To deliver a dial, the DO needs row 3.

This is not the TCP blackhole case, where the connection is dead and everything stalls. Here TCP is genuinely fine and the protocol ping is succeeding. The health check is correct. It is answering the wrong question.

The fix

The fix is an application-level round trip, where the pong is produced by your own code:

async webSocketMessage(ws: WebSocket, msg: string | ArrayBuffer) {
  const role = this.roleOf(ws);

  if (role === "agent") {
    // Control frames on this socket are text. Do not hand a non-string to
    // JSON.parse. `JSON.parse(typeof msg === "string" ? msg : "")` throws
    // SyntaxError on the empty string — ask me how I know.
    if (typeof msg !== "string") return;
    try {
      const control = JSON.parse(msg) as { op?: unknown };
      if (control.op === "ping") {
        // The point is that THIS code ran. That is the signal.
        safeSend(ws, JSON.stringify({ op: "pong" }));
      }
    } catch {
      // Drop malformed control frames silently. A heartbeat that becomes a new
      // reason to disconnect is worse than no heartbeat.
    }
    return;
  }

  this.forward(role, msg);
}

The machine sends {op:"ping"} every 45s and reconnects after two missed cycles (100s, so two periods plus slack).

Do not reach for setWebSocketAutoResponse() here:

// This puts the blind spot right back.
this.ctx.setWebSocketAutoResponse(
  new WebSocketRequestResponsePair("ping", "pong")
);

That API exists specifically to answer without waking the DO, which makes it exactly wrong as a liveness probe, for the same reason protocol ping was: the reply proves the edge is up, not that your object still holds the socket.

Shipping the heartbeat before the Worker

We shipped the machine agent with the new heartbeat while the old Worker was still deployed. The old Worker ignores {op:"ping"}, so no pong ever arrives. An agent that treats silence as death then severs a perfectly healthy connection every 100 seconds and keeps reconnecting until the Worker rolls out.

The guard is one clause:

bool controlHeartbeatExpired({
  required bool pongSeen,
  required Duration sinceLastInbound,
}) => pongSeen && sinceLastInbound > kControlPingTimeout;

pongSeen && reads as redundant: if it timed out, surely it is dead. Delete it and you get the reconnect storm above, but only during the window where client and server versions disagree. Staging, where you deploy both together, will not reproduce it. We pinned it with a test so nobody deletes it for looking useless.

If your client and server deploy independently, decide explicitly what happens when the peer does not answer your new protocol.

What protocol ping is still good for

We did not remove it. Data connections still run pingInterval = 15s.

Mobile NAT reclaims idle mappings within tens of seconds, and after that nothing arrives and nothing errors. A terminal just sits on its last frame forever. Edge-proxied pongs detect this perfectly well, because if TCP were actually dead the proxied reply could not reach you either.

Data connections kept pingInterval = 15s. Only the control socket got the application round trip.

This came out of building Otter Beam, which lets you reach the tmux sessions on your own machine from a phone. The relay only forwards the encrypted SSH stream; it cannot read it.