During a payments incident in August I had our error rate panel on the big screen, showing the last day. There was a clear bump at a quarter past two. I dragged across it to zoom in on the twenty minutes that mattered, and the line disappeared. Not to zero. No data at all, for exactly the window we were trying to understand. When I zoomed back out, the bump came back.
The panel divided the rate of our 5xx counter by the rate of all requests, both over a range of $__interval. That is a Grafana variable that grows and shrinks with the time range, so a panel showing a day does not ask for thousands of points. Over a day it came to two minutes. Over twenty minutes it would have been about two seconds, so Grafana raised it to the minimum interval of the data source, which takes the scrape interval configured there. Nobody had configured one, and the default is fifteen seconds. Our Prometheus scrapes that service every sixty.
rate needs at least two samples inside its range to calculate anything. A fifteen second window over sixty second scrapes holds one sample at most, so the query returned nothing for every step, at precisely the zoom level an incident asks for. We had been reading that panel for more than a year. On an ordinary day nobody zooms in.
The fix to the query was one word. $rate_interval is never less than four times the scrape interval, so the window always holds enough samples. The data source now has its scrape interval set to the sixty seconds we actually use, because Grafana has no other way to know it. A linter in the dashboards repository fails any query that puts $interval inside rate, irate or increase, and on its first run it found forty three more. The ratios we alert on are now recording rules with a fixed five minute window, so a panel and the alert behind it read the same series, and each panel states the window in its title.
A blank panel in the middle of an incident looks like a broken tool, and people stop looking at it. Ours was broken only at the scale we needed it, and it had been since the day it was built.
– Sergey Shinder
