Support kept forwarding the same complaint: the site hangs for half a minute, several times a day, usually mid-morning. Every graph we opened was flat. Latency looked fine, error rate looked fine, saturation looked fine, and we spent two weeks quietly suspecting the customers' networks.
They were right and our instruments were not. Our scrape interval was sixty seconds, the alert expression used rate(...[5m]), and the alert had for: 5m on top of that. A thirty second period of total saturation, spread across a five minute rate window and then required to persist for five more minutes, arrives at the evaluator as a bump of a few percent. It was never going to fire. Worse, the dashboards people trusted were built on the same expressions, so the investigation and the alerting shared a blind spot.
The floor on what you can see is set by the whole chain: scrape interval, rate window, evaluation interval, and the for duration multiply into a minimum incident length. Ours was somewhere around four minutes of sustained badness. Anything shorter was arithmetically invisible, and most of what annoyed our users was shorter.
The fix was not simply shortening windows, which just buys noise and cardinality. We changed what we alert on. Symptom counters cannot be averaged away: a request either exceeded two seconds or it did not, so we alert on the count of slow and failed requests over a short window with no smoothing, and page when the count crosses a number rather than when a smoothed rate crosses a percentage. Trend dashboards keep the five minute rates, because that is what they are good for.
For the resources that actually spike, we record max_over_time on a fifteen second series, so a graph shows the peak rather than the average of the peak and the quiet around it.
The first day the new rule was live it fired three times before lunch, each time for about forty seconds, each one matching a support ticket. Ask what duration of badness your monitoring is mathematically unable to report, because your users are already experiencing it.
– Sergey Shinder
