Jev: one judgment call, or twelve dimension scores? I measured both on three classification tasks

The obvious way to use Jev (TypeSafe AI's System One judgment API) is one question per row, then threshold the score. I wanted to know whether the other shape earns its extra tokens: 12 to 14 narrow questions per row, cached scores, weights fitted locally on my own labels. Three tasks, both shapes, identical rows: 25,174 API calls, 5,477 test rows, 34.1M input tokens, $1.43.

The two shapes

Jev takes a text plus a list of questions and returns a calibrated 0–1 score per question. It is $0.042 per 1M input tokens, output free, ~32k of context, and it reports its own confidence per question.

• Direct judgment. One question per row, then threshold the score.
• Feature extraction. 12–14 questions per row in one request, raw scores cached to JSONL, an L2 logistic regression fitted on my own labels; multi-class is one-vs-rest plus argmax.

I wanted the second shape for operational reasons first: extraction costs money and needs the network, so it caches and resumes; fitting is local and free, so I can change weights, thresholds, or the whole stack without spending another token. Worth nothing unless the decomposition beats one well-written question.

Experiment one: a task my own question could ace

200 synthetic rows of B2B replies built from 32 template families, deliberately poisoned: "yes" rows containing "we'll pass", "no" rows containing "absolutely" and "happy to meet".

The direct question scored 100%, and swapping the choice names also scored 100%: no position effect. The noul form at a 0.5 threshold got 96.0%. My decomposed pipeline scored 98.0%, losing four rows and winning none. The test was worthless: the answer sits in front of one question. Those 200 rows cost 292,924 input tokens in 5.7 seconds, zero failures.

The split that turned 98% into 90%

That 98.0% (AUC 0.998) came from a random 5-fold split. Holding out whole template families instead of single rows dropped the same features to 90.0% (AUC 0.970): that apparent skill was the model recognising phrasing it had already been trained on.

A character-bigram naive Bayes on the same grouped folds scored 93.5% (AUC 0.989); a word-level one scored 47.5%. My API-scored dimensions lost to a free model whose sibling was worse than a coin flip.

Two rules stuck: print the majority baseline before believing any number, and choose the split — grouped by family or generator, never random rows — before building the dataset.

Experiment two: a task where one call should fail

300 rows, 150 per class, no names, no keyword, no single sentence that decides the label; the label exists only as a residue of many weak stylistic cues across the whole text. I handed the Jev call both class style descriptions, so it knew what to look for.

Direct choice: 64.7%. Choice order swapped: unchanged. The noul form scored 62.0% (AUC 0.676).

Dimensions plus fitted weights, 5-fold out-of-fold: 74.0% (AUC 0.811). McNemar exact on the same rows with a paired bootstrap of 20,000 resamples: +9.0 points over direct choice, 95% CI [+3.0, +15.0], 57 wins against 30 losses, p=0.0050; +11.7pt over noul, CI [+5.0, +18.3], p=0.0012; +7.7pt over the single best dimension, p=0.0140.

I reimplemented the cross-validation in plain numpy, not trusting my own code: 73.7% (AUC 0.809). And the CLI prints the in-sample score beside the out-of-fold one — 100% adjacent to 74%.

Confidence came apart from correctness

On 126 of the 300 rows (42%) the Jev call reported confidence of 0.9 or higher, and on exactly those rows it was 72.2% accurate. Confidence and correctness separated where the task got hard.

The obvious ensemble failed too: the Jev call's probability as a 13th input beside the 12 dimensions scored 73.0% against 74.0% — its information was already inside the dimensions.

Then a character-bigram naive Bayes on the raw text, same folds, scored 74.0% (AUC 0.824) — the same number as the Jev-scored dimensions, for zero API calls.

The real ledger export broke the script twice

Third task: my own bookkeeping export. 4,907 train rows, 2,101 test rows, labelled with the debit account and restricted to the top 12 debit accounts, which cover 87.4% of rows (about 55 accounts exist in total). The input was the description text, the amount, and the credit-side account with its sub-account; I dropped the debit sub-account because it gives the label away.

One direct question carrying 12 answer choices scored 0.3998: it reads the words correctly and still cannot choose among twelve labels when asked once. Comprehension was never the bottleneck; the output shape was.

14 dimensions with fitted weights: 0.9105. Word-bigram naive Bayes: 0.9491, beating both by +3.9pt over the dimensions (p<1e-4). Their errors differed, though: n-grams alone rescued 138 rows, the dimensions alone 63. Stacking the 12 n-gram class probabilities beside the 14 dimension scores reached 0.9695 — +2.33 points over n-grams alone (95% CI [+1.57, +3.14]) and +5.90pt over dimensions alone. The question was never which model wins, but whether they are wrong on the same rows.

For a detector, the headline should be false positives

339 rows of hard benign text: text that mentions injection techniques while being entirely benign — security write-ups, red-team notes, a post like this one. The single Jev question flagged 1.5% of them; my 12 dimensions flagged 37.2%; word-bigram naive Bayes flagged 43.4%. The gap between 1.5% and 37.2% is 35.7 points (p<1e-4) — about 25× the review load. The external set runs the other way: the direct call held 0.9558 on 2,060 rows where the dimension model managed 0.8398 and the n-gram baselines collapsed to 0.35–0.38, because the labels there run against the surface.

Four fixes, four failures

(a) Add hard benigns to training — 170 rows: 37.2% → 34.3%, detection held at 0.9554. Barely moved. (b) Drop the surface-level dimensions: 40.7%, worse. (c) Set the threshold on fold A for roughly 5% FPR: fold B landed at 8.3%, but attack detection fell from 0.9522 to 0.770, an 18-point loss. (d) Feed the Jev call's probability in as a 13th feature: benign FPR 46.0% and the external set fell from 0.9558 to 0.7883.

The fitted weights explain it: the two largest positives were obfuscated_encoding (+1.80) and hidden_in_content (+1.62), both high for an attack string and for a security document that quotes one. This is a dimension-design problem, not a weights problem: none of my 12 questions asks whether a text uses the technique or merely discusses it, and that is the dimension to write next.

The per-source table taught me the most

On competition-style attack strings the single Jev call collapses to 0.665 while the dimensions hold 0.977. On wildguard_judgecomp every method sits between 0.63 and 0.67 — a shared ceiling, not a method difference, so I stopped treating that slice as evidence. Across the test set, dimensions beat the single Jev call by +11.6pt (p<1e-4) but only tie word bigrams (+0.2pt, p=0.90).

Japanese NLI, where decomposition finally earned its keep

8,000 train, 2,434 test rows, three classes one-vs-rest, neutral 55% of the test set. Every n-gram baseline landed at or below majority: 0.5534, premise-only word bigram 0.5423, word bigram 0.5251, character bigram 0.4996. The hypothesis-only probe, the standard label-leak test, scored 0.5394 — so no label is hidden in the hypothesis.

14 dimensions reached 0.9076 (macro-F1 0.8845) against 0.8373 for the single Jev call (0.8198): +7.03 points, 95% CI [+5.67, +8.38], p<1e-4, with 234 rows solved only by the dimensions and 63 only by Jev. Leak check two: zero fully duplicated premise+hypothesis pairs between train and test, 21 duplicated premises only.

The single Jev call stalls at 0.675 in the 0.60–0.80 Jaccard band while the dimensions get stronger as overlap rises (0.987 in the 0.80+ band). The gap is all in contradiction (0.921 against 0.654); on entailment the single Jev call wins, 287/353 against 271/353. The labels are noisy — reverse-direction pairs carry different labels (jne-00015, jne-00016) — so 0.9076 is accuracy against human labels, not a ceiling.

Nine minutes, 25,174 calls, one error — mine

All extraction for all three tasks — 8 runs, 25,174 calls, 20,139 rows — finished between 22:59:51 and 23:09:05 JST: nine minutes and fourteen seconds, zero failed calls. The 4,391 injection training rows took 76 seconds; 8,000 JNLI rows 129; 4,907 ledger rows 83.

The one error was mine: at 23:06:24 a question object named rel_choice was missing its instructions field, and the CLI refused it with exit code 1 and the field name rather than judging something I had not asked. A criteria field passed alongside noul is the same class of usage error; a missing dimension stops with exit 1 and names it, while --allow-missing mean-imputes it.

Two things came from experiment, not documentation: noul points the way I did not expect (a true statement scored 0.96, a false one 0.02, so it is the probability the proposition is true), and --out merges rather than appends — re-running the same file reported skipped=300 / extracted=0, so a long job can resume without re-paying.

The bill is the cheap part

34.1M input tokens: 27.9M for feature extraction ($1.17) plus 6.23M for direct questions ($0.26) — $1.43 at $0.042 per 1M input tokens. Output was 846,765 tokens, counted separately and not priced.

Per row, one direct question costs 444–619 tokens, 12 dimensions 1,010–1,036, and 14 dimensions 1,136–1,154. Each dimension adds 40–50 tokens, mostly the wording of the question, so twelve dimensions run 1.6–2.3× one question, and fourteen run 1.8–2.6×: $0.019–0.026 per 1,000 rows against $0.042 and $0.048, or $19–26 per million rows against $42 and $48. Extraction is paid once: 4,391 training rows cost $0.19, and every later fit or threshold change reuses that cache for nothing.

At a million rows a month with 20% benign traffic, a 37.2% false-positive rate sends about 74,000 rows to human review; at 1.5% it sends about 3,000. That is roughly 71,000 rows — about 590 hours at 30 seconds per row — against a monthly token bill of $42 versus $19–26.

The ladder I actually run

Guardrails get one direct call with a strict threshold plus a small dimension model consulted where that call is unsure; classification gets n-grams first, with dimensions stacked on the residual.

What I got wrong, and the counterexample I still want

In order of how much it cost me: believing a random-split CV; treating a judge's confidence as calibrated; testing on data my own generator had made too easy; and reporting accuracy for a detector whose real risk is false positives.

What I still want is a counterexample: some task where the dimensions beat a direct call that was already competent — what did they know that one question could not? I have not found one in three tasks and I would like to be wrong.

The API used here: TypeSafe AI's Jev (System One, jev-latest) — text plus a list of questions in, a 0–1 score per question out, own confidence included, roughly 32k of context, $0.042 per 1M input tokens with output free. Each dimension costs 40–50 tokens, mostly the wording of the question; that is where the 1.6–2.3× comes from.

Two caveats: I hand-wrote all 12–14 dimensions, so these numbers describe my dimension design as much as feature extraction; and the ledger task covers only the top 12 account classes and took the credit-side account as input — the import-with-context case, not a bare bank statement. The JNLI labels carry noise and a 55% neutral skew, the external injection set has contradictory labels on near-identical rows, and output tokens are not priced.

The full tables and threshold sweeps are at https://agentjournal.dev/blog/llm-judge-vs-feature-extraction/.