Checkout latency at the ninety ninth percentile sat around nine hundred milliseconds, and the CPU graph for those pods never went above forty percent. We spent a fortnight on the wrong problem because of that graph. Slow code on idle hardware sends you looking at locks, at the database, at anything except the machine that is clearly not busy.
A CPU limit in Kubernetes is not a speed limit in any sense that matches the word. It is a quota per accounting period. Our limit of five hundred millicores meant fifty milliseconds of CPU time in every hundred millisecond window. The service is written in Go, and the runtime sized its parallelism from the node, which has sixty four cores, so it happily ran a dozen goroutines at once. Twelve threads spending fifty milliseconds of quota between them exhaust it in about four milliseconds of wall clock. For the remaining ninety six milliseconds of that window the whole cgroup is not scheduled at all. Not slow, not preempted, not running.
Averaged over a minute, that is forty percent utilisation, which is exactly true and completely useless. The request that arrives at millisecond five of a window waits for the window to end before it does anything, and nothing in a utilisation graph has the resolution to show that.
The counter that shows it is the throttled seconds total, which we had never put on a dashboard. Those pods were being held for about sixty milliseconds of every hundred, hour after hour, for as long as the metric had existed.
Three things changed. The runtime now takes its parallelism from the cgroup limit rather than the node, so the work spreads across the period instead of burning it in a burst. Latency sensitive services kept their requests and lost their limits entirely, which was an uncomfortable conversation and the right outcome; batch work kept limits, because for batch a throttled period costs nothing. And every service dashboard carries a throttling panel next to CPU, with an alert when more than one period in twenty is capped.
Utilisation is an average taken over a second. Throttling happens inside a hundred milliseconds, and that is where the customer is waiting.
– Sergey Shinder
