For most of a Wednesday afternoon our search API had a p99 of exactly two and a half seconds. Not two point four, not two point six. A flat line, perfectly level, for three hours, while support forwarded complaints about searches that took twenty seconds or never came back at all. The objective was three seconds. As far as every alert we owned was concerned, we were comfortably inside it.
The latency histogram used the default buckets from our client library, whose largest finite boundary is ten seconds, but somebody had trimmed the list two years earlier to cut series count and the largest boundary we kept was two point five. Above that there is only the infinite bucket. When more than one percent of requests land beyond the last finite boundary, the quantile function has nowhere to interpolate and returns the highest boundary it has. Our p99 was a ceiling that looked like a measurement.
It had been the ceiling for two years. Nobody noticed because on a normal day the p99 sat around four hundred milliseconds, far below it, and the chart looked like data. It only turned into a lie at exactly the moment we needed it, when the tail went past the instrument's range.
Access logs carry the duration, and a query over them put the real p99 for that afternoon at seventeen seconds.
The buckets are now chosen per service, from the objective outward: dense around the threshold we alert on, and extending past the longest timeout anywhere in the call path, so a request cannot be slower than the histogram can express. Services with a wide spread use native histograms, which do not need boundaries picked in advance. Every latency panel shows, beside the quantile, the share of requests in the infinite bucket, and an alert fires if that share goes above a tenth of a percent, since it means our numbers have stopped meaning anything. We also added the ratio of requests slower than three seconds as a direct count, which is the objective itself and needs no interpolation at all.
An instrument has a range. Past the end of it, it does not report the truth or an error. It reports the end, and a quantile that cannot go higher will look calm during every incident large enough to matter.
– Sergey Shinder
