One of the crates classifies a failure into its worst matching category. The category list is an array where each entry carries a code, and a check prints how much of that list the classifier actually covers.
OK_WORST_CLASS arms=13/13
Thirteen of thirteen, every run, for as long as the check had existed. I read that as good news for months. It was not news at all.
The numerator was built out of the denominator
Reduced to the shape, the check was this:
const ARMS: [Arm; 13] = [ /* ... */ ];
let codes: Vec<Code> = ARMS.iter().map(|a| a.code()).collect();
println!("OK_WORST_CLASS arms={}/{}", codes.len(), ARMS.len());
codes is ARMS with a function mapped over it. map does not drop elements. So codes.len() is ARMS.len() by construction, and the fraction is n/n for every possible value of n, including zero.
This is not a subtle bug in the counting. There is no input, no state, no ordering that makes this print anything except full. The check had no failure mode. A check with no failure mode is a println! wearing the costume of a check.
The first repair was also wrong
The obvious fix is to make the numerator mean something. So the numerator became "arms whose code agrees with what the classifier returns for that arm", which is a real property that can be false.
The print went to arms=13/13 again. This time it deserved the number, and it was still not measuring the thing anyone was afraid of.
The fear is not "an arm I wrote disagrees with itself". The fear is an error case with no arm at all. A new variant gets added to the error type. Nobody adds a row to ARMS, so the classifier silently files it under a neighbour.
| what the fraction can see | what it cannot |
|---|---|
| an arm whose code disagrees | a case with no arm |
| an arm that was written wrong | a case nobody wrote |
| drift inside the list | absence from the list |
Both halves of the fraction were still derived from ARMS. A list cannot report its own omissions. Whatever is missing from it is missing from the numerator and the denominator at the same time, which is exactly the cancellation that keeps the ratio at 1.
The denominator has to come from the other side
The repair is not in the counting. It is in where the denominator is read from.
// before: both sides from the list of arms I wrote
arms = agreeing_arms / ARMS.len()
// after: the denominator is the thing being covered
arms = agreeing_arms / ErrorKind::ALL.len()
ErrorKind::ALL is the closed set the classifier is supposed to cover. Now adding a variant without adding an arm moves the denominator and not the numerator, and the print falls to 13/14 on the commit that introduces the gap.
The general rule, which is obvious once stated and easy to violate once writing code:
A coverage number is only meaningful when its denominator comes from a different source than its numerator. Same file is a warning. Same array is a guarantee of nothing.
The class is findable by machine
The part I find most useful is that this does not require noticing. An audit can read a check's own source and ask whether the expression producing the numerator is derived from the expression producing the denominator. If it is, the check gets a named finding. "Aggregation gap" is now a category in that audit, sitting next to the other ways a check lies.
It fired on two checks the day it was written, and both had been green since the day they were merged. Neither had ever been suspected, because a green check is not something anyone goes looking at.
$ audit --class=aggregation-gap
FAIL worst_class numerator derived from denominator (map over the same array)
FAIL arms_declared denominator is a compile-time constant of the numerator's source
Two things I keep
A ratio that has never moved is not evidence of health. It is an untested hypothesis about your own arithmetic, and the longer it has held at 100% the less anyone remembers to doubt it. The question to ask a green check is not "is this right" but "what input makes this print something else".
Absence is the thing coverage is for, and absence is exactly what a self-derived denominator cannot represent. If the list defines both what you have and what you should have, you have written a tautology and given it a percent sign.
