TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision boundary. In a simulation where the judge is a clean, unbiased, well-behaved instrument, overall agreement of 0.92 comes with agreement of 0.60 on the examples within 0.05 of the gate's threshold. Nothing is wrong with the judge. The summary statistic is answering a question you are not asking.
Every judge validation writeup I have read, including two of my own, reports agreement as a scalar. We sampled n examples, humans labelled them, the judge labelled them, they agreed 87 percent of the time, ship it.
The number is real. It is also computed over a population that is mostly irrelevant to what the judge is for.
The mismatch, stated plainly
If you use a judge to gate, the judge is a classifier with a threshold. Its job is to sort examples into pass and fail. Almost all of its value, and all of its risk, lives in a thin band around that threshold, because that is the only region where a small error changes an outcome.
An example whose true quality is far above the cut gets passed by any judge that is not broken. An example far below gets failed by any judge that is not broken. Those examples cost you nothing when the judge is noisy, and they are the overwhelming majority of your validation set. So they set your headline agreement number, and they tell you almost nothing about the decisions you actually delegated.
This is not a claim about any real judge. It follows from the geometry.
A simulation, and what it is and is not
I want to be careful about what follows, because a number that reproduces from your own simulation is not thereby right. What is below is a simulation. It measures no real model. It is a demonstration that a mechanism exists and is large, not evidence about the size of the effect in your system.
The construction is deliberately friendly to the judge. Latent quality is uniform on 0 to 1. The human label is the ground truth by definition: pass if quality exceeds 0.5. The judge sees quality plus symmetric Gaussian noise and applies the same 0.5 threshold. So the judge's latent score is unbiased and, near the threshold, calibrated. It has no position bias, no verbosity bias, no prompt sensitivity, and agrees with the human criterion exactly. Its only flaw is noise.
I hold that construction fixed and sweep one parameter, the noise scale.
import numpy as np
def run(sd, n=200_000, thr=0.5, seed=0):
rng = np.random.default_rng(seed)
q = rng.uniform(0, 1, n) # latent quality
human = q > thr # ground truth by construction
judge = (q + rng.normal(0, sd, n)) > thr # quality plus noise, same threshold
agree = human == judge
out = {"overall": agree.mean()}
for lo, hi in [(0.0, 0.05), (0.05, 0.15), (0.15, 0.30), (0.30, 0.50)]:
band = (np.abs(q - thr) >= lo) & (np.abs(q - thr) < hi)
out[f"{lo:.2f}-{hi:.2f}"] = agree[band].mean()
return out
for sd in [0.05, 0.10, 0.15, 0.20, 0.30]:
print(sd, {k: round(v, 3) for k, v in run(sd).items()})
Rows are the noise scale. Columns are distance from the threshold.
judge noise sd 0.05: overall 0.960, within 0.05 of threshold 0.685, 0.05 to 0.15 0.958, 0.15 to 0.30 1.000, 0.30 to 0.50 1.000
judge noise sd 0.10: overall 0.919, within 0.05 of threshold 0.600, 0.05 to 0.15 0.827, 0.15 to 0.30 0.981, 0.30 to 0.50 1.000
judge noise sd 0.15: overall 0.879, within 0.05 of threshold 0.566, 0.05 to 0.15 0.740, 0.15 to 0.30 0.925, 0.30 to 0.50 0.994
judge noise sd 0.20: overall 0.840, within 0.05 of threshold 0.549, 0.05 to 0.15 0.686, 0.15 to 0.30 0.864, 0.30 to 0.50 0.973
judge noise sd 0.30: overall 0.772, within 0.05 of threshold 0.532, 0.05 to 0.15 0.627, 0.15 to 0.30 0.773, 0.30 to 0.50 0.904
Read the second row. Overall agreement is 0.919, which is the number that goes in the writeup and gets called strong. In the band where the gate is actually deciding, it is 0.600, against a coin-flip floor of 0.500.
Read the Overall column downward, then the near-threshold column. As the judge gets noisier, the headline degrades gently, from 0.96 to 0.77. The near-threshold column has already collapsed by the second row and then barely moves, because it is pinned near 0.5. Once noise dominates the signal at the boundary, more noise cannot make it much worse. The headline keeps a lot of room to look respectable while the only region you care about has been at chance for some time.
Why the headline is so insensitive
I have made a version of this argument before, about how an aggregate pass rate hides the slice that matters. The difference here is that the slice is not one I picked. It is wherever your threshold happens to sit.
The bands are not equally populated. With uniform quality, examples within 0.05 of the threshold are 10 percent of the set. The outer band, 0.30 to 0.50 away, is 40 percent, and agreement there is 1.000 in the two lowest-noise rows.
So the reported scalar is roughly a weighted average in which the hardest decile carries a tenth of the weight and the trivial 40 percent carries four times that. If your true quality distribution is not uniform but concentrated near the threshold, which is what I have watched happen once a team has been optimising against the gate for a while, the picture gets worse rather than better, because the population moves into the band where the judge is weakest.
This is a cousin of a known problem, not the same one
The classic reference for "a single agreement summary can mislead" is Feinstein and Cicchetti, High agreement but low kappa: I. The problems of two paradoxes, J Clin Epidemiol 1990;43(6):543-9, doi:10.1016/0895-4356(90)90158-L. Worth reading if you have not.
I want to be precise about the relationship, because they are not the same mechanism and I do not want to borrow authority I have not earned. Their paradoxes are about marginal imbalance: with skewed marginal totals, a high raw agreement can produce a low kappa, and asymmetry can raise kappa relative to symmetry. That is a property of the fourfold table.
What I am describing is different. It is heterogeneity of agreement across the latent scale, where the summary is fine as arithmetic and simply averages over a region that matters and a region that does not. Both land in the same practical place, which is that one number is not enough, but for different reasons, and a fix for one will not fix the other.
What to report instead
Three changes, in increasing order of effort.
- Report agreement in bands of distance from the threshold, not just overall. This costs one group-by and it is the whole finding. If you only do one thing, do this.
- Report the population of each band. The near-threshold share tells you how much of your traffic the judge is genuinely deciding, and it moves over time as your system improves and as your team optimises against the gate.
- Sample your validation set by band rather than uniformly. If you are going to spend human labels, spend them where the judge is uncertain. Uniform sampling buys you a precise estimate of the agreement rate on examples whose outcome was never in doubt. I have argued for stratifying eval samples before, on score class. The band is a different axis, and the reason I prefer it is that it falls out of the gate's geometry rather than out of somebody's judgement about which classes are interesting.
The second one is the one I underrate. A judge validated when 8 percent of traffic sat near the threshold is not validated for a system where 30 percent does, and no property of the judge changed in between.
FAQ
Does this mean judge validation is useless?
No. It means the scalar is a summary of the wrong population for a gating decision. The same labels you already collected can be re-cut into bands at no extra labelling cost.
Is this just the same as reporting precision and recall instead of accuracy?
Related but not the same. Precision and recall also aggregate over the whole set. You can have good precision, good recall, and near-chance agreement in the boundary band, because that band is small.
My judge outputs a discrete 1 to 5, not a continuous score. Does this apply?
Yes, and it is harder to see, because the band structure is coarse. Use the distance from the cut in score units, and if your gate is "3 or above", the 3s are your near-threshold band. They are usually the largest single bucket and the least reliable.
Does a stronger judge model fix it?
Not across the range I swept. A stronger judge is a smaller sd, which is the top row rather than the bottom, and agreement within 0.05 of the threshold there is still 0.685 while the headline reads 0.96. Push sd small enough and the gap does close, and it is not even monotone: extending the sweep, the gap between overall and near-threshold agreement peaks around sd=0.10 at 0.32 and falls to 0.04 by sd=0.005. But sd=0.05 is already a very quiet judge and the gap there is 0.28.
Should I move the threshold to where the judge is confident?
That is choosing your decision boundary to flatter your instrument rather than to match your risk. If you genuinely have freedom in where the cut goes, set it on cost, then measure agreement there.
Open question
The band analysis tells you where the judge is unreliable. It does not tell you what to do with those examples at run time.
The obvious move is routing: when a score lands within some distance of the threshold, send it to a human, or to a second judge, or to a stronger model. But the distance is measured in the judge's score space, and the judge's score near the boundary is exactly the quantity we just established is noise-dominated. So the router is deciding what is uncertain using the measurement whose uncertainty is the problem.
I do not have a clean answer. Judge-reported confidence is one candidate and I distrust it for the usual reasons. Disagreement across a small ensemble is another, and it costs what it costs. If you have shipped a boundary router that survived contact with production, I would like to know what you keyed it on.
