Docker never restarted my database: the price of restart: no

Something died in the middle of the night. I found out in the morning, from a user, not from a machine.

The container had exited. It stayed exited. Docker had watched it die and done nothing, precisely because I had told it to do nothing. The line responsible was three characters long, repeated on every service in my compose file.

This is part 5 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the difference between a container that crashes and a system that recovers.

The line, on all seven services

restart: no

Every service had it: nginx, the API, Postgres, Redis, MinIO, the dashboard build, the workspace helper. Seven out of seven.

restart: no is Docker's default, and it means exactly what it says. If the process exits, for any reason, the container stays dead. There is no supervisor, no retry, no backoff. It also means something worse that I had not thought through: if the VPS reboots, nothing comes back up. The machine boots, the Docker daemon starts, and every container sits there stopped, waiting for me to SSH in and run make up. A host reboot after a kernel update would have taken the whole product offline until I noticed by hand.

For a laptop, restart: no is the sane default: you do not want yesterday's experiment relaunching itself when you reboot. On a production server, it is a decision to have no automatic recovery at all.

The part that surprises people: I had healthchecks

Here is what makes this worth writing about. It was not that I had no monitoring at the container level. Two services had real, well-written healthchecks:

db:
  healthcheck:
    test: ['CMD-SHELL', 'pg_isready -U ${DB_USER} -d ${DB_NAME}']
    interval: 10s
    timeout: 5s
    retries: 5

api:
  healthcheck:
    test: ['CMD', 'bun', '-e', "fetch('http://localhost:8000/health')..."]
    interval: 30s
    timeout: 10s
    retries: 3
    start_period: 40s

And the API even waits for the database properly, which is the right pattern:

api:
  depends_on:
    db:
      condition: service_healthy    # not just "started"

So why did none of that help? Because of a fact that catches almost everyone coming from Kubernetes:

In plain Docker Compose, a failing healthcheck does not restart anything. It flips the container's status to unhealthy and stops there. There is no liveness probe semantics, no automatic replacement. Unlike a Kubernetes liveness probe or Docker Swarm, Compose has no built-in actor that reacts to unhealthy.

So a healthcheck without a restart policy produces a container that is beautifully, accurately labelled as broken, that nobody restarts and nobody is watching.

Two mechanisms, two different failure modes

Once you see it this way, the design becomes obvious. They are not alternatives, they cover different failures:

I had the second half of the second row missing, and the entire first row missing. Which is how a crash at 3am became a morning outage.

Choosing a restart policy

There are four values, and only one is usually right for a production service:

unless-stopped is the one you want for long-running services. It restarts the container on crash, brings it back after the host reboots, and still lets you deliberately docker compose stop a service without Docker second-guessing you the next time the daemon starts. always looks similar but will resurrect a container you intentionally stopped, which is maddening during an incident.

db:
  restart: unless-stopped
api:
  restart: unless-stopped
nginx:
  restart: unless-stopped

Note that this applies to long-running services only. One-shot containers (a build step, a migration runner, the workspace helper that just runs an install) should stay no, because restarting a job that already completed is not recovery, it is a loop.

Making "unhealthy" actually do something

Restart policies handle crashes. They do nothing for the nastier case: the process is still running, so Docker is happy, but it stopped serving. That is what healthchecks detect, and in Compose you need to add the actor yourself. Two honest options:

• A small autoheal sidecar: a container with access to the Docker socket that watches for unhealthy status and restarts those containers. It is a handful of lines in the compose file and it closes the gap without introducing an orchestrator.
• External monitoring that alerts you, so a human decides. Slower, but it also catches the failures a restart cannot fix (a full disk, a dead dependency, an expired certificate).

The second one is not optional even if you add the first, because "restart it" is not always the right answer, and a service that silently restart-loops all night is its own kind of outage. Getting that visibility is the next article in this series.

While you are in there, close the smaller gaps too. In my file, only the database and the API had healthchecks: Redis, MinIO and nginx had none. And nginx used the short form of depends_on, which only waits for the other containers to have started, not to be healthy:

# before: nginx starts as soon as api exists, ready or not
nginx:
  depends_on: [storage, api]

# after: nginx waits until they can actually serve
nginx:
  depends_on:
    api:
      condition: service_healthy
    storage:
      condition: service_healthy

What this does not fix

Restart policies keep you up through crashes and reboots. They do not give you zero-downtime deploys. My deploys still run docker compose up -d --force-recreate, which stops the container and starts a new one, so every deploy is a few seconds where requests hit nothing. That is a different problem with a different fix (build the image elsewhere, then swap), and it belongs with the deploy pipeline, alongside the day my build filled the disk.

Being honest about the boundary matters: unless-stopped turns "down until I wake up" into "down for a few seconds". That is a huge win, and it is not the same as high availability. On one VPS, there is still exactly one of everything.

The takeaway

restart: no is the right default on a laptop and a decision to have no recovery on a server. If your production compose file has it, a crash at 3am is an outage until you notice, and a host reboot is a full outage until you SSH in.

Set unless-stopped on every long-running service. Give every service a healthcheck, not just the database. Use condition: service_healthy in depends_on so things start in an order that actually works. And remember the part that trips people up: in Compose, a healthcheck only labels the problem. Something else has to act on it, whether that is an autoheal sidecar or an alert that reaches you.

Originally published on jguillaumesio.com