Our configuration service was tier three. No SLO, no pager, a single replica for a long time, and a line in the architecture document describing it as a soft dependency because clients cache their values and can carry on if it is unavailable. That sentence was true about the steady state and completely false about the one moment that mattered.
It went down at 09:40 during a routine node pool rotation, which meant pods were also being rescheduled across the fleet. A pod that restarts has an empty in-memory cache. The client library fetched configuration during application startup, blocking, retrying for two minutes, and then failing the process. Readiness never passed. Kubernetes restarted the pods, they blocked again, and over about twenty minutes the rotation walked through the cluster and took down services that had nothing to do with configuration and were themselves perfectly healthy.
Nothing about this was subtle in hindsight. Any dependency your process cannot start without is a hard dependency, whatever tier you have filed it under, and a cache that only exists in memory does not survive the exact event that empties it.
We fixed the client first. The last known good configuration is written to a file on disk and read at startup, and the defaults are baked into the image at build time as the final fallback, so a cold pod with no network can still come up serving the values it had yesterday. Startup no longer blocks on the fetch; it starts with what it has and refreshes in the background, and it exports a metric for the age of its configuration so that stale-but-running is visible rather than silent.
The service itself is now tier one with three replicas and a pager, because pretending otherwise did not make it less important.
The habit that came out of this is the one I would recommend. Once a quarter, in staging, we block a dependency at the network level and restart everything that claims not to need it. The first run found three more services that could not start without something described somewhere as optional.
You do not know what your dependencies are. You know what your dependency diagram says, and it was drawn by someone who has not restarted the system with that box removed.
– Sergey Shinder
