Our readiness probe froze the whole API: an async def with a synchronous connect inside

The short version

When docker compose ps says the api container is unhealthy, is it right? And what is
it actually checking?

Our health checks turned out to have one probe that checks too hard, one that checks too
lightly, one that checks the wrong place, and one setup that never receives the
configuration you think you gave it:

1. Too hard. The vector-store readiness probe is an async def that makes a synchronous gRPC connection. With the vector store absent, every probe freezes the API process's entire event loop for 10+ seconds (10.1 s measured locally; the whole request took 34 s inside the container last month).
2. Too lightly. Object storage is a hard readiness gate, but it is only really checked until the first success. After that the result is cached for the life of the process. If MinIO dies after startup, readiness keeps saying connected.
3. Wrong place. The production compose file probes /api/v1/health/ready. That path does not exist; it returns 404. The endpoint lives at /health/ready.
4. Missing config. Start the stack without --env-file .env, and every key in .env that also appears in the compose environment: block is silently replaced by the compose default. Keys that do not appear there pass through fine.

The rest of this post walks through each layer. The last section is a troubleshooting
table you can keep open next to a terminal.

1. Three endpoints, three jobs

The health routes live in server/app/api/v1/health/router.py and are mounted without a
prefix (server/app/main.py:429: app.include_router(health_router, tags=["health"])),
so the paths are exactly what they look like:

A liveness endpoint that checks nothing is correct: it answers "is the process alive?", and
tying it to dependencies would get the process restarted every time a dependency blips.

The readiness checks themselves are clear (router.py:82–106):

try:
    await db.execute(text("SELECT 1"))
    db_status = "connected"
except Exception:
    raise HTTPException(status_code=503, detail="Database is unavailable")

try:
    await storage.ensure_ready()
    storage_status = "connected"
except Exception:
    raise HTTPException(status_code=503, detail="Object storage is unavailable")

try:
    await vector.check_ready()
    vector_status = "connected"
except Exception:
    vector_status = "unavailable"

Database and storage gate readiness; the vector store is reported but does not gate. The
docstring's reason is that non-vector endpoints keep working without it, so there is no
point pulling the instance out of rotation (router.py:74–77). That design is sound. The
problems are in how each gate is implemented.

2. Who checks whom in compose

In the development stack, docker/docker-compose.yml:

Three one-shot containers (minio-init, migrate, bootstrap) run with restart: "no",
and downstream services wait for them with condition: service_completed_successfully,
i.e. for a zero exit code.

The chain: api waits for postgres, redis, minio, milvus and vault to be healthy and for
all three one-shots to succeed (docker-compose.yml:264–280). web waits for exactly one
thing: api healthy (:298–300).

So the api probe is the choke point. If it fails, the frontend never starts.

3. The symptom we already knew about

We hit this last month while writing up a minimal demo topology, so briefly: drop milvus
and etcd from the stack, and /health/ready still returns the correct
"vector":"unavailable" — after 34 seconds. The api probe allows 3 s
(urlopen(..., timeout=3)) inside a 5 s compose timeout (docker-compose.yml:282–284).
34 against 3 fails every time, so api stays unhealthy, and web, which depends on
api: service_healthy, never starts. The workaround we gave —
docker compose up -d --no-deps web — still works.

We also wrote: "api showing unhealthy is expected; the service itself is fine."

That was half right.

4. For those seconds, the whole API is frozen

Here is the vector probe (server/app/adapters/vector/milvus.py:112–115):

async def check_ready(self) -> None:
    """Probe vector-store connectivity; raise if unreachable (for readiness checks)."""
    self._ensure_connected()
    utility.get_server_version()

It is an async def, but _ensure_connected() (:89–99) calls connections.connect(...),
a synchronous pymilvus call. Synchronous network I/O inside a coroutine means the event
loop can do nothing else until it returns.

I pulled that out and ran it locally (soit/server/.venv, pymilvus 2.5.11): connect to a
port nobody listens on, with a second task on the same loop recording a timestamp every
0.5 s.

probe failed: MilvusException
probe 10.1s, max gap between 0.5s ticks 10.3s, ticks during probe 0

In 10.1 seconds, the ticker did not run once. The 10.1 s is pymilvus's own default
connect timeout; the 34 s in the container also included resolving a service name that did
not exist on the Docker network.

Two probes back to back, to see whether a failure is remembered:

probe 1 failed: MilvusException
probe 1 10.2s
probe 2 failed: MilvusException
probe 2 10.1s

It is not. A failed connect leaves no connection behind, connections.has_connection("default")
is still false next time, and every probe pays the full price again.

Now put that back into compose. The development api runs a single-process uvicorn
(docker-compose.yml:261, no --workers), and the healthcheck fires every 10 s.

🔍 This part is inference, not something I measured inside a container: each probe
freezes that one event loop for 10+ s (34 s in the container), while compose gives up on
the probe after 3 s and sends the next one 10 s later — before the server has finished
being stuck on the previous one. In a topology without a vector store, this API process
spends most of its time stuck inside its own health check. The login that succeeded
last month either landed between two probes or waited in line for one to finish. "The
service itself is fine" — but only in the gaps.

Two other adapters in the same repository get this right:

• The pgvector backend's check_ready is await asyncio.to_thread(self._check_ready) (adapters/vector/pgvector.py:100–102): the blocking call goes to a thread and the loop keeps running.
• Every synchronous storage operation goes through _run_sync_operation, which wraps asyncio.to_thread in asyncio.wait_for (adapters/storage/fsspec.py:23–35), with a 10 s default (settings.py:178, storage_operation_timeout_seconds).

So the fix is not a mystery: give the Milvus probe the same to_thread + wait_for
treatment with a one- or two-second budget. That is exactly what issue #44 proposes; the
issue just saw "slow", not "frozen".

5. The opposite problem: a gate that only checks once

Storage is a hard gate, so you would expect it to be the strictest. Here is ensure_ready
(adapters/storage/fsspec.py:120–158):

async def ensure_ready(self) -> None:
    if self._ready:
        return
    async with self._ready_lock:
        if self._ready:
            return
        ...
        exists = await _run_sync_operation("storage_ready", ..., self.fs.exists, readiness_root)
        ...
        if not exists:
            raise KernelError("STORAGE_NOT_READY", "Storage root is not ready", ...)
        self._ready = True

After the first success _ready is True, and every later call returns on the first line.

How long that lasts depends on how long the adapter instance lives. Readiness gets it from
the DI container's get("storage_port") (router.py:44–47), and the container's get()
caches whatever a factory builds as a singleton (wiring/container.py:350–372). So
within one API process, the storage gate only really checks until it first succeeds.

A local reproduction with fsspec's memory:// backend, no external services involved:

1st probe: ok
root removed, exists = False
2nd probe on same instance: ok (cached)
fresh instance raised: KernelError STORAGE_NOT_READY Storage root is not ready

The root is gone, the same instance still reports ready, and only a fresh instance notices.

That is useful at startup: api will not report ready before MinIO is up. But if MinIO goes
away later, /health/ready keeps returning "storage":"connected" while real reads and
writes fail one by one. The name ensure_ready is honest — it means "make sure we are
initialised", not "probe whether it is reachable right now". The problem is that readiness
uses it as the latter.

The existing unit test (tests/unit/test_health_readiness.py:52–57) covers the 503 branch
with a fake store that raises immediately; it does not cover "worked, then went away".

6. The production probe asks for a path that does not exist

The development probe hits /health/ready. Production,
docker/docker-compose.production.yml:151–156:

healthcheck:
  test: ["CMD-SHELL", "python -c \"import urllib.request;urllib.request.urlopen('http://localhost:9200/api/v1/health/ready')\""]
  interval: 15s
  timeout: 10s
  retries: 5
  start_period: 30s

There is an extra /api/v1. As section 1 showed, the health router has no prefix. Hitting
the current application directly with Starlette's TestClient:

/api/v1/health/ready 404 {"success":false,"code":"NOT_FOUND","message":"Not Found",...
/health/live 200 {"success":true,"code":"OK","message":"OK","data":{"status":...

urlopen raises HTTPError on a 404, the probe exits non-zero, and the api container in
the production stack stays unhealthy permanently.

It has not broken production only because nothing there waits on api's health: the gateway
and the frontend use the short depends_on: - api form
(docker-compose.production.yml:116–118, :166–167), which waits for the container to
start and ignores health. So the probe fails silently — an unhealthy in docker ps, and a
permanent alert in any external monitor that trusts container health.

Two more things while we are here:

• The production probe passes no timeout to urlopen (the dev one passes timeout=3), so only compose's 10 s stands behind it.
• Readiness is not reachable from outside either: the Caddy gateway sends /api/* to api and everything else to the frontend (docker/production/Caddyfile:18–19). By those rules, an external load balancer asking for /health/ready lands on the web container.

Two docs also use the prefixed path (docs/QUALITY_GATE.md:199,
docs/operations/database-connections.md:74); the quickstart has it right
(docs/quickstart.md:76). I plan to open an issue covering all of these.

7. Half of your .env is ignored

This is the first pitfall listed in issue #27, and it deserves its own section because the
symptom is not "it failed to start". It is "it started, with somebody else's config".

Every server container in the dev compose has two config sources
(docker-compose.yml:159–215):

env_file:
  - path: ../.env
    required: false
environment:
  DATABASE_PASS: ${DATABASE_PASS:-soit}
  SECRET_KEY: ${SECRET_KEY:-change-me}
  ...

Two compose rules combine badly:

1. When a key is set in both environment: and env_file:, environment: wins.
2. ${DATABASE_PASS:-soit} in environment: is interpolation, and interpolation reads the .env in the project directory (here docker/, where the compose file lives) or the file given with --env-file — not the ../.env that env_file: points at.

So with a .env at the repository root and no --env-file, ${DATABASE_PASS:-soit} finds
nothing, falls back to soit, and overrides the value env_file just loaded.

docker compose config proves it without starting anything. Root .env:

DATABASE_PASS=from-root-env
MY_ONLY_KEY=only-in-env-file

--- without --env-file:
      DATABASE_PASS: soit
      MY_ONLY_KEY: only-in-env-file
--- with --env-file .env:
      DATABASE_PASS: from-root-env
      MY_ONLY_KEY: only-in-env-file

Same file: one key applied, one silently swapped for the default. The environment:
block lists 51 keys — database, Redis, MinIO, Vault, SECRET_KEY, every model provider's
API key — and all of them behave this way. Keys that are not listed there, such as
MILVUS_MODE or VECTOR_BACKEND, pass straight through.

The worst part is that everything looks fine. The postgres container's POSTGRES_PASSWORD
falls back to the same default, both sides agree on soit, the database connects, the
stack comes up — and the password and SECRET_KEY you set were never used.

The quickstart command includes --env-file .env (docs/quickstart.md:13), so copying it
is safe. Editing the command, or running docker compose up from inside docker/, is not.

8. Milvus Lite is not a container workaround

Early in September the repo gained MILVUS_MODE=lite: an embedded Milvus Lite engine on a
local file, no milvus or etcd containers. It looks like the answer to section 3. It is not,
for this compose file:

• It is a single-process file store. The commit message says it plainly: one process owns the database file and nothing else can read it.
• In compose, knowledge-ingest-worker writes vectors and api queries them — two containers, no shared volume. By construction, each would open its own file.
• Production refuses lite outright (settings.py:550–554).

It is a local debugging switch for the knowledge base (see the Milvus Lite section of
docs/development.md), not a way to deploy without Milvus.

9. Troubleshooting table

10. What still does not line up

As usual, our own findings first.

① The vector probe makes a synchronous connect on the event loop, with no timeout. Each
probe freezes the whole process for 10+ s. Issue #44 records "slow"; I plan to add the
"frozen" reproduction there. Fix: to_thread + wait_for, same shape as the storage
adapter.

② The storage gate only checks until the first success. If storage disappears after
startup, readiness does not show it, so an orchestrator will not pull an instance whose
storage is gone. Workaround: monitor object storage directly rather than trusting api
readiness. I plan to open an issue.

③ The production api probe gets a 404 and has no timeout. The production api is
permanently unhealthy and external monitors alert forever. I plan to open an issue.

④ Two docs use a readiness path with a prefix that does not exist
(QUALITY_GATE.md:199, operations/database-connections.md:74). Same fix as ③.

⑤ The production gateway does not expose readiness. By the Caddyfile's routing rules,
/health/ready lands on the frontend, so an external load balancer has no readiness URL
to use.

⑥ The ReadyResponse docstring says status can be not_ready (router.py:31–32),
but the code never returns it; when not ready it raises a 503. Small, but anyone writing a
client from the docstring will check a field value that never appears.

⑦ knowledge-ingest-worker and scheduler have no healthcheck. A hung process still
shows Up.

⑧ Without --env-file, .env keys that share a name with the environment: block are
silently overridden. That is documented compose behaviour, not a bug, but the way our
compose file is written makes it very easy to hit, and nothing reports it. I plan to make
it the first entry in the troubleshooting doc that issue #27 asks for.

One more that I did not verify — I only noticed its absence: the database engine
configures no explicit connect timeout or pool-wait timeout
(infra/db/session.py:65–73 passes only pool_pre_ping, pool_size and max_overflow),
so how long the database gate waits when the host is unreachable is up to driver defaults.
I did not measure it and am not drawing a conclusion.

Caveats

• This time things were run, but nothing ran inside a container. The four local runs: pymilvus connect time and event-loop blocking, the storage cache (memory:// backend), TestClient against two paths, and docker compose config for interpolation. The code is soit main at commit 8a24af4.
• The 34 s is last month's in-container measurement; "the API is frozen most of the time in the container" is inference from a single-process uvicorn plus the locally measured loop blocking. I did not measure request queueing inside a container.
• Your number will differ from my 10.1 s. That is pymilvus's default connect timeout; an unresolvable name, a firewall dropping packets and a closed port all take different amounts of time.
• I only read the Community edition.
• This corrects a sentence in our own previous post. "The service itself is fine" was based on a successful login. The observation was right; the conclusion stopped one step short.
• Disclosure: I maintain SOIT.

The one-line takeaway

A health check is not boilerplate. What a probe checks, how often, how long it takes, and
whether it slows down the thing it is checking all need to be verified separately — and
healthy only means "the last probe process exited with 0".

Try it, and tell me where I am wrong

The repository is github.com/soit-ai/soit. Three quick
places to check:

1. server/app/adapters/vector/milvus.py:112–115 next to adapters/vector/pgvector.py:100–102 — look for to_thread in each async def;
2. server/app/adapters/storage/fsspec.py:120–158, starting at if self._ready:;
3. the probe path at docker/docker-compose.production.yml:152 next to how the router is mounted at server/app/main.py:429.

If you hit a symptom that is not in the table, please open an issue — the troubleshooting
doc in issue #27 is still unclaimed, and your pitfall might be its next entry.