Measure the Judge Before You Trust It: Self-Consistency Comes Before Human Agreement

Here's a question almost no eval pipeline can answer: if you asked your LLM judge to score the exact same response five times, would you get the same number back?

Most teams never check. They wire up an LLM-as-judge, pick a threshold that feels right ("block anything under 7"), and ship it as a merge gate. Then they spend months wondering why the same PR is green on one run and red on the next. The judge wasn't wrong. It was unstable, and nobody measured the stability before trusting the number.

I built maf-evals — a three-tier agent evaluation reference on Microsoft Agent Framework and .NET 8 — partly to force this discipline into the open. The rule that fell out of it: measure the judge before you trust it, and measure self-consistency before you even think about human agreement. Every number below came from actually running the calibration command, not from reasoning about it.

The two questions, in the only order that works

When you calibrate a judge, there are two questions:

1. Does the judge agree with itself?
2. Does the judge agree with a human?

The second question is meaningless without the first. If a judge gives the same input a 5, then a 2, then a 4, then a 5, then a 2 — comparing its "score" to a human label is comparing to a coin flip. You can't calibrate a ruler that changes length every time you pick it up.

So maf-evals runs them in order. Twelve hand-labelled cases, each judged three times, self-consistency first.

dotnet run --project src/EvalRunner -- calibrate --repeat 3

Self-consistency: where a comfortable average hides a broken metric

Here's the self-consistency table from an actual run. Three RAG-triad scores — Retrieval, Groundedness, Relevance — each judged three times across the calibration set:

Look at Retrieval. A mean standard deviation of 0.20 looks fantastic. If you stopped there — as most dashboards do — you'd conclude the judge is rock solid and promote it to a blocking gate.

That average is lying to you. Given the same input five times, Retrieval returned 5, 2, 4, 5, 2. Most cases were perfectly stable, which dragged the mean SD down to a cozy 0.20 — while two cases swung a full three points. The consequence is the column that actually matters: 17% of cases would flip a merge decision at random.

Think about what that means in CI. Nearly one in five pull requests, near the threshold, gets a merge verdict decided by nothing but which sample the judge happened to draw. That's not a quality gate. That's a random number generator with a nice UI.

The fix isn't to tune the prompt until Retrieval calms down. The fix is a doctrine: a score that moves between runs has no business blocking a merge. Retrieval was demoted to advisory only. The actual gating for "did the right documents come back?" is done by an exact, free, deterministic check — comparing returned chunk IDs against expectedChunkIds. No judge, no wobble.

This is the core independence idea in the whole repo. Evidence ranks on an independence axis — independent to corruptible — not a cost axis. Rules gate; judges report. A deterministic ID comparison is unforgeable and never flips, so it gates. A judge's opinion moves between runs, so it reports.

Only now do you ask about human agreement

Groundedness and Relevance passed self-consistency (0.00 SD), so they've earned the right to be compared against human labels. Here's that table:

Now look at Groundedness, and notice the trap. Its bias is -0.17 — practically zero. If bias were your health metric, you'd call it well-calibrated and move on.

Groundedness is broken in two opposite directions at once. It scores outright fabrication at exactly 3.0, every single time — and it separately penalises well-grounded answers for being slightly off-topic. Those two errors point in opposite directions, so they cancel in the aggregate, and the bias reads as a healthy -0.17. The metrics that expose the truth are mean absolute error (1.17) and band agreement (75%) — not the average.

Which is why the Groundedness floor moved from 3.0 to 3.5. At a 3.0 floor, every hallucination — parked at exactly 3.0 — slipped through as a mere warning instead of a block. That single threshold change lifted band agreement from 50% to 75%.

That's the whole point of calibration: the threshold wasn't chosen by taste. It was chosen because the labelled fixtures showed hallucinations clustering at 3.0, so the floor had to sit above them.

Two thresholds, because judges wobble and rules don't

Because judge scores move between runs, a single cut-off turns every borderline case into a coin flip — exactly the 17% problem. So in maf-evals, every judge score gets two thresholds:

• a floor that blocks, and
• a target that warns.

The band between them absorbs the wobble. Deterministic checks — chunk-ID matches, tool-call comparisons, rule evaluations — get no band. They always block, because they don't move.

Conceptually, the gate logic looks like this:

public GateOutcome ApplyScore(JudgeScore score, ScoreBand band)
{
    // Deterministic checks never reach here — they block or pass, no band.
    if (score.Value < band.Floor)
        return GateOutcome.Block(
            $"{score.Name} {score.Value:F1} < floor {band.Floor:F1}");

    if (score.Value < band.Target)
        return GateOutcome.Warn(
            $"{score.Name} {score.Value:F1} < target {band.Target:F1}");

    return GateOutcome.Pass();
}

// Groundedness after calibration: floor raised so hallucinations (which
// the judge parks at exactly 3.0) fall below it and BLOCK.
var groundedness = new ScoreBand(Floor: 3.5, Target: 4.0);

The floor at 3.5 isn't a guess. It's the number the labelled data demanded, verified by the jump in band agreement.

The rule that keeps you honest

One more constraint makes this whole thing trustworthy: re-run calibration after you change the judge model, the labelling guide, or any threshold. Scores from different judges are not comparable. A 3 from gpt-4o and a 3 from next quarter's model are different quantities wearing the same label. If you swap the judge and keep the old thresholds, you've silently thrown away the calibration and you're back to picking numbers by feel.

And the calibration set itself is built so the three scores pull apart from each other — twelve cases engineered so a judge can't get a good grade by collapsing retrieval, groundedness, and relevance into one vague sense of "quality." If your calibration cases don't isolate the failure modes, your calibration proves nothing.

The takeaway

An LLM judge is a measuring instrument, and you would never trust a measuring instrument you hadn't checked against a reference. Yet teams routinely promote a raw judge score straight to a merge gate.

The order is non-negotiable:

1. Self-consistency first. If the judge disagrees with itself, nothing else matters. Watch the worst-case range and verdict-flip rate, not the mean SD — a comfortable 0.20 average hid a 17% coin flip.
2. Human agreement second, and only for scores that survived step one. Watch MAE and band agreement, not bias — Groundedness's near-zero bias hid a metric broken in two directions.
3. Anything that wobbles gets demoted from gate to advisory, and the deterministic check does the real gating.

Rules gate; judges report. Measure the judge before you trust it. If a score moves between runs, it doesn't belong on the merge path.

The full three-tier implementation — guardrails inside the agent, the PR gate, trajectory judging, plus the calibration harness and an adversarial safety suite — is on Microsoft Agent Framework and .NET 8 here: github.com/sauravbhattacharya001/maf-evals. If you're about to trust an LLM judge with a blocking decision, run calibrate --repeat 3 on your own judge first. The number that comes back will change how you gate.