At 03:40 on a Wednesday, seven pods belonging to four unrelated services were killed on the same node. The status on all of them read Evicted, the message said the node was low on resource: ephemeral-storage, and none of those four services had done anything at all that night.
Our catalogue export job had. It streams a partner feed, transforms it, and uploads the result, and it does the middle part in a file on an emptyDir volume. On an ordinary night that file is about three gigabytes. The partner had sent a full re-export rather than a delta, and the file grew to forty-six. The node had a hundred gigabytes of disk, shared between container images, container logs and every emptyDir on it.
When the kubelet crosses its eviction threshold it starts removing pods, and the order it removes them in is the part we had not thought about. Pods are ranked by their usage of the pressured resource relative to what they requested, and a pod that requested nothing is treated as being over its request from the first byte. We had set memory and CPU requests on everything, carefully, from real observed usage. Not one workload in the cluster declared an ephemeral-storage request. So the ranking was close to arbitrary and our smallest, most innocent pods went first.
Image garbage collection ran at the same time, as it is supposed to, and deleted the layers the evicted pods needed in order to restart, which is why recovery took eleven minutes rather than one.
Four changes. Every emptyDir now has a sizeLimit, which turns this into the offending pod being evicted alone rather than the node running out. A LimitRange sets a default ephemeral-storage request and limit for any pod that does not declare one. The export job writes to object storage in chunks and keeps nothing large on local disk, which is where it should have been from the start. And node disk usage is alerted on at seventy-five percent with a panel breaking it down per pod, so the growth of that file is visible for hours before it matters.
We had been thorough about two resources and had not noticed there was a third. The node shares out whatever nobody has claimed, at the moment it can least afford to.
– Sergey Shinder
