From twenty to eight on a Monday, every quote our pricing service produced came back with tax at zero. Thirty four minutes, about nine hundred quotes, a few dozen of them accepted by customers at a price we then had to correct by hand. Production only. Staging was perfect all morning, and so was the same image running on my laptop.
The service ships its reference data inside the image, in a directory called reference: tax tables per country, rounding rules, a list of jurisdictions. It is read once at startup. On the Friday we had added a persistent volume to the chart so that an operator uploaded rate override file would survive a restart, and the mount path in the values file was reference, because that is where the overrides go.
A volume mount is not a merge. Kubernetes does not add the volume's contents to what the image has at that path. It puts the volume there instead, and everything the image shipped in that directory stops existing from the container's point of view. The volume was empty, because it was new. So the loader found no tax tables, and the loader had been written to treat an empty reference directory as no rates configured, which it reported as a single info line and a zero rate rather than as a reason not to start. The readiness probe checks that the HTTP server answers. It did.
Staging was fine because persistence is disabled there by a flag, and the flag exists because nobody wanted to pay for volumes in an environment that gets rebuilt nightly.
Three things changed. The reference directory is read only in the image and an admission policy rejects any pod mounting a volume underneath the application directory, which is the change that makes this class of mistake impossible rather than unlikely. The loader now fails startup if the table set is empty or older than the image it came in, and an empty rate table cannot produce a price at all. And the override file, which was the actual requirement, lives in its own directory with its own mount and is layered over the shipped tables in code, where it can be reasoned about.
A mount path is a statement about what the container is allowed to see. Ours quietly said: not the numbers we charge tax with.
– Sergey Shinder
