Jev is getting wired into agents everywhere as the cheap typed-judgment layer (Choice/Score/Boolean with probabilities). Every leaderboard ranks judges by accuracy. But accuracy isn't what your agent acts on — the probability is. Nobody was checking whether those probabilities are honest. So I built the harness to do it.
Judge Audit (https://github.com/kunko-ai-labs/judge-audit): an open harness that audits whether an AI judge's probabilities mean what they say — ECE, reliability diagrams, accuracy-coverage curves, cost/latency, drift. Judge-agnostic by design (jev / llm / simulated adapters); Jev is just the first target because it's the one people are actually deploying.
Three audits, every number recomputable from raw per-row checkpoints in docs/runs/:
- Clean emails (200): 200/200, ECE 0.0036, min confidence 0.89. Suspiciously perfect — synthetic and easy, published as a calibration baseline, not a production claim. So I built a malicious set.
- Adversarial emails (200): prompt injection, cyrillic homoglyphs, zero-width chars, fullwidth, synthetic PII, social engineering. 95.5% overall, ECE 0.0389 (10x worse than clean). Injection fooled it 7/40 — but mean confidence collapsed 0.996 → 0.71 under attack. In this dataset, failures often coincided with lower confidence: a usable abstention/escalation signal, not a guarantee. Homoglyphs/zero-width/fullwidth/PII: 100% on the tested cases. Social engineering: 0% attacker success.
-
Task router (120): easy→cheap model, hard→frontier. It said
route_easy120/120 times. Dijkstra, N-queens, knapsack → cheap model at 0.98–1.0 confidence. Hard accuracy 0%, ECE 0.93 on the hard slice. Confidently, systematically wrong — the exact failure mode no accuracy leaderboard would catch.
Caveats, because I know this crowd: the router ground truth is by construction (task difficulty as designed), not empirically validated against cheap vs frontier models — that's the next experiment. The 0% attack success on cost-inflation is an artefact of the never-route-strong bias, not robustness. All flagged in the reports.
v0.3.0 ships an MCP server (pip install 'kunko-judge-audit[mcp]', tools: run_audit, check_drift, list_judges). Next: Jev vs OpenJev vs Claude vs a local open model on the same datasets, side by side — tell me what you'd add to the matchup.
Apache-2.0, Sigstore-attested artifacts. Brutal feedback welcome, especially on methodology.
24-second video built from the real router numbers: https://github.com/kunko-ai-labs/judge-audit/releases/download/v0.3.0/brag.mp4
