The last post ended on a promise: background workers fall into the same correlated-failure trap as request services, from a different angle. This is that angle, and it starts one level earlier: a health check that reports green while the work has stopped.
I floated a heartbeat file for a queue worker, a small marker the worker rewrites as it goes, with a watchdog checking how stale it is. The reply came back fast: "just put a /health on it." Fair reflex. It's what every request service does, what every tutorial shows, what every load balancer expects. It's also the one thing that hides the failure I was trying to catch. An HTTP /health on a worker will return 200 all day while the worker sits there doing nothing.
Process is up, actual work has stopped. That gap in between is what the whole post is about.
Where the HTTP reflex comes from
"Health check means an HTTP endpoint" is baked in for a good reason. Every service that handles requests exposes one, every orchestrator polls one, every tutorial you've read wires up GET /health returning 200. For a request handler that's not just convention, it's correct. The reflex only misfires when you carry it, unexamined, onto a process that doesn't handle requests at all. Same shape as the session library from the last series entry: a default that's right in the context it was built for, and surprising the moment you step outside that context. Nobody was wrong to reach for it. It's answering a question the worker doesn't ask.
The failure the green endpoint can't see
Picture the worker. It runs a loop: pull a job off the queue, do the work, pull the next one. To satisfy the "just add /health" reflex, you bolt an HTTP listener onto it, and that listener answers independently of the loop, from its own thread or process or whatever your runtime uses.
Now the loop wedges. A job deadlocks, the connection pool is exhausted and every task is stuck behind one that never returns, a lock is held by a thread that's gone. Timeouts help, and most libraries ship with them, but a default one isn't a tuned one, so the loop can sit far longer than the work really needs before anything gives up. The work stops. The listener doesn't: it keeps accepting connections and returning 200, because answering a trivial GET has nothing to do with whether the loop is moving.
So the orchestrator sees green. The queue behind the worker climbs. And the exact failure you put the health check there to catch, a worker that's alive but stuck, is the one failure the check is structurally blind to. The endpoint is answering exactly the question it was asked, and that question was about the HTTP server, not the loop.
Proof of life isn't proof of work
The obvious patch is to make the check smarter. The HTTP /health only proves the listener answers, fine, so have it also confirm the worker process is actually running: check the PID, ask the supervisor, systemctl is-active. Not hard to wire up. And still the wrong signal, because a dead process is often not the failure you're actually worried about. Far more often the process is up and the loop is parked on a job that will never return.
The signal has to come from the work
The reason the HTTP reflex is right for a request service and wrong for a worker is the same fact read two ways.
In a request service, handling the request is the work. When the health check hits an endpoint and gets a response, the act of responding is itself proof that the request path is alive: routing, framework, handler, all of it just ran. The check can't easily lie, because doing the thing it checks is how it passes.
A worker breaks that coupling. The work is the loop, and an HTTP listener bolted on beside it answers independently of the loop, so by default the check proves the listener is up, not that the loop is moving. Nothing stops you from wiring the two together: have the loop update a shared timestamp and let /health return 503 when that timestamp goes stale. Do that and the endpoint is honest again. HTTP can carry a real signal; it only has to originate in the loop, something only the loop can refresh as it makes progress, and stop the moment the loop stops. Where you expose it, a file, a metric your monitoring already scrapes, an HTTP route, a socket, matters less than whether it's tied to the work.
A heartbeat the loop writes, a watchdog that reads it
The simplest thing with that property: the loop writes a small marker after it does real work, and a separate watchdog checks how stale the marker is.
# inside the worker loop
while true; do
job="$(reserve_job)" # blocks until there is work
process "$job"
touch /run/worker.alive # heartbeat: proof the loop advanced
done
A file under /run or /dev/shm carries real weight here: the marker means something precisely because only the loop can refresh it. A separate listener can't fake it, because it never touches the loop.
The watchdog runs on cron, out of process, and does one thing. If the marker is older than a threshold, mark the instance unhealthy and let the platform replace it.
# watchdog on cron: replace the instance if the loop has gone quiet
threshold=$(( max_job_seconds + slack )) # longest healthy job, plus margin
age=$(( $(date +%s) - $(stat -c %Y /run/worker.alive) ))
if (( age > threshold )); then
mark_instance_unhealthy # let the platform replace it
fi
Then the naive version bites. Look at where the touch sits: after process. The marker only advances when a job completes. Now think about an empty queue. reserve_job blocks, nothing arrives, the loop never reaches the touch, and to the watchdog that's indistinguishable from a loop wedged on a stuck job. Both stop refreshing the file. A worker that's perfectly healthy and simply out of work gets cycled for no reason. This assumes reserve_job returns on a timeout: it long-polls for a few seconds, then comes back empty. If instead it blocks forever on an empty queue, the loop stalls there, the marker goes stale, and even the fixed heartbeat won't save an idle worker from being killed. The poll has to be able to time out for any of this to hold.
So the marker should prove one thing: the loop is still turning. Move the heartbeat so it fires on every iteration, idle poll included, and pick the threshold above your longest legitimate job so a slow-but-healthy job isn't read as a wedge:
while true; do
job="$(reserve_job)" # blocks up to a poll timeout, then returns
[ -n "$job" ] && process "$job"
touch /run/worker.alive # proof the loop turned, job or no job
done
That fixes the single-threaded loop. But most real workers aren't single-threaded, and that's the next assumption to pull out. They run a pool, a handful of threads or goroutines or async tasks pulling jobs in parallel, and one touch from a shared spot lies about all of them. Say ten workers share the process and one wedges on a deadlocked query while the other nine keep churning through small jobs. The marker stays fresh, the instance stays green, and you've quietly lost ten percent of your throughput with a connection stuck open the whole time. That's a grey failure, and a central heartbeat is exactly blind to it, for the same reason the HTTP check was: it's measuring something beside the work, not the work. The signal has to come from all of them at once. Each one refreshes its own timestamp and the watchdog reads the oldest live one, or the pool supervisor checks its children before it touches the file.
Whether the loop is doing useful work (draining a queue that's actually backing up) is a different question, and liveness is the wrong place to answer it. That belongs to queue-depth monitoring, one level up. Keep the two apart, or you'll wire queue depth into liveness and rebuild the exact correlated-failure trap from the last post, this time on your workers. And even a healthy heartbeat only proves the loop turns, not that the turning gets anything done, which is a separate failure this post doesn't close.
A few more details bite. First, startup: a worker that just booted hasn't turned the loop yet, so the marker is missing on a healthy instance. Seed it at boot, or give the watchdog a grace window that covers boot plus the first poll cycle before it's allowed to judge a fresh instance. Otherwise every deploy shoots its own new workers. And the threshold cuts both ways: it has to clear your longest honest job, so if that job runs forty-five minutes, a genuinely wedged worker also sits forty-five minutes before the watchdog notices. Long jobs and fast recovery pull against each other here, and the way out isn't a bigger threshold, it's the job itself reporting progress as it runs, a different signal living one level up.
Second, restart versus replace. The tempting fix when the watchdog fires is to bounce the worker in place through the process supervisor: SIGTERM, wait, then SIGKILL if it's still there. The pragmatic move is to skip the ceremony and just rotate the whole instance. It's simpler and clears more, a wedged socket, a leaked handle, memory corrupted into a corner, all gone with the box. What it won't clear is a cause that travels with the work: a job that wedges whatever picks it up, or a shared dependency that's still down. But those don't survive an in-place restart either, so there's nothing to weigh. Rotating is slower, so wire the infra to bring the replacement up before it pulls the wedged one out, unless you like explaining capacity dips.
Why put HTTP on a worker at all
Once the signal is a file the loop touches, a separate HTTP endpoint mostly earns its keep on a request service, where the listener is already there and a /health route is free. A worker usually has no listener at all, so bolting one on means a server, a port, and sometimes a second container in the pod whose whole job is to answer 200. If the file already carries the signal, that's a lot of machinery to expose the same fact.
On Kubernetes you can skip the listener entirely. The worker doesn't sit behind a Service, so there's no traffic to gate with a readiness probe, which leaves liveness as the whole question here (the last post pulled those two apart). Nothing routes to an HTTP port anyway, and the liveness probe reads the marker directly with an exec:
livenessProbe:
exec:
# probe runs with no shell, so wrap the test yourself
command: ["sh", "-c", "[ $(( $(date +%s) - $(stat -c %Y /run/worker.alive) )) -lt 60 ]"]
periodSeconds: 10
failureThreshold: 3
# slow first turn? use a startupProbe, not a fat initialDelaySeconds
Two catches with the exec approach. First, the image has to carry sh, stat and date, and carry the flags the command expects, which a distroless image won't and a busybox one bends, so a probe that passes on your laptop can misfire on Alpine. Verify it against the real base image, or ship a tiny static helper that does the check and exits non-zero. Second, exec isn't free: every probe forks three processes, so ten seconds a pod across a big fleet is a steady drip of spawns and kubelet overhead.
There's a way around both, and it's where the signal being a metric pays off. If the loop already exports its heartbeat as a metric your monitoring collects, whatever you run, an alarm defined on that metric, going stale past a threshold, can fire the reaction from outside the pod: no forks, no shell, no exec on the box at all. The shift is that this detects and alerts rather than restarts, the alarm pages someone or triggers automation, where a liveness probe would have the kubelet kill the pod on the spot. So it catches the wedge but doesn't reset it by itself. Pair it with a probe if you want the automatic restart, or run it alone where a human or a separate automation closes the loop. A file with exec is simplest and self-contained, an in-process handler on a shared timestamp skips the fork tax, and a metric moves the check off the box entirely at the cost of that directness.
On EC2 with an ASG the file wins more easily, because a watchdog already runs on the box reading that same marker. A cron job comparing the marker's age to a threshold is enough; systemd has a native watchdog that does the same thing if you'd rather not run cron. Either way, an HTTP endpoint on top buys nothing the watchdog doesn't already have. I haven't run workers on ECS myself, so I'll leave its specifics alone rather than guess, but the split is the same wherever you land: the signal comes from the loop, not the transport.
Long-running processes don't get an honest signal for free
The worker is one case of a wider shape. Crons, queue consumers, stream processors, anything long-running that isn't already sitting behind something that checks it, none of them get an honest health signal for free the way a request handler does. And the failure follows a similar pattern every time: a green endpoint, a running process, an exit code of zero, standing in for work that quietly stopped. The transport says alive. The work says nothing, because nobody wired the work to speak.
The fix generalizes as cleanly as the trap does. Put the signal somewhere only real work can emit it, and when it stops arriving in the window you expect, rotate the process. Whether the loop is doing anything useful beyond staying alive, draining a backlog, actually committing rows, is a second signal living one level up, and worth building separately. Get the first one right and a stuck worker stops looking healthy. Get both and you can see it slow down before it stops.
A worker wedged on a dead job used to sail through its health check and pile work up behind a cheerful 200. Wire the signal to the loop, and it trips the check the moment it stops moving, which is the moment you wanted to know.
Two things this leaves open, both about the same blind spot. A worker can turn its loop forever without making progress, and it can hang precisely because it's beating on a dependency that needs room to recover. Same root: a signal that says "alive" over a process doing no useful work. Which of those bites you first decides what's worth building next.
