Jev vs a 310M encoder I trained myself: 750 rows, three tasks, two different winners

TL;DR: I ran three Japanese classification tasks (250 rows each, same gold labels) through six systems. A 310M encoder I fine-tuned on 250 labels beat TypeSafe's Jev on topic classification by +12.0 points (McNemar p=0.00007) and was 4–20× faster — but on the two polarity tasks it only tied Jev. The zero-shot open-source models everyone is writing about lost everywhere: a GLiNER-family classifier scored 62.8 / 87.2 / 44.8 against Jev's 76.8 / 94.4 / 74.0. The same architecture family, trained on my labels, scored 88.8 / 92.8 / 75.2.

One-line summary: A decision API wins when you have no labels; a small trained encoder wins when you have a few hundred — and which one wins on a given task is decided by the shape of the task, not by which model is trending.

The question behind the hype

Jev (TypeSafe AI's System One) is genuinely good, and it is cheap: $0.042 per 1M input tokens with output free, roughly 32k of context, a confidence number in the response, and no training data required. If you read the last two weeks of posts about it, the conclusion is "use Jev."

I use Jev. But I had a pile of results from my own tasks that did not fit that conclusion, and one of them was embarrassing enough that I re-ran it twice:

• A char n-gram + logistic regression baseline beat Jev on long-document topic classification by 11.6 points (0.884 vs 0.768).
• The same baseline lost to Jev by 20.8 points on short-text polarity (0.736 vs 0.944).

A 5-minute linear model beating a decision API on one task and losing badly on the next is not a story about either model. It is a story about task shape. So I designed the experiment I should have run first: hold the answers fixed, hold the rows fixed, and separate the two axes that actually differ — labels or no labels, and topic or meaning.

What I measured, and how

Three Japanese datasets, 250 rows each, frozen:

Six systems, two conditions:

• Zero-shot (0 labels): Jev, a local 26B MoE (Gemma 4 26B-A4B, Q4_K_M, resident on the same mini-PC), SemIf (Qwen3.5-4B), GLiClass multilang-mini (a GLiNER-family classifier), Laya (typed decisions).
• Trained (250 labels): sbintuitions/modernbert-ja-310m with a classification head, and a char n-gram + logistic regression baseline.

The trained numbers are 5-fold cross-validated out-of-fold predictions: train on 200, predict the held-out 50, five times, so all 250 reported predictions come from a model that never saw that row. That is the same protocol I use for the linear baseline, which means the trained row and the baseline row are directly comparable.

Everything is compared on identical row ids with McNemar's test. This matters: at n=250 the independent-confidence interval is about ±6 points, so "0.888 vs 0.944" cannot be called a difference without a paired test.

Results

McNemar, same ids:

Finding 1: the same family wins or loses depending on whether you trained it

This is the part I did not expect, and it is the most useful thing in the table.

GLiClass is a GLiNER-family classifier — same lineage as the models currently being pitched as open-source Jev alternatives. Zero-shot, it scored 62.8 / 87.2 / 44.8: it lost to Jev on all three tasks, and on the 3-class polarity task it was barely above chance (44.8% against 33.3%).

Trained on 250 of my labels, the same generation of model scored 88.8 / 92.8 / 75.2. The gap between zero-shot and trained is +26.0 / +5.6 / +30.4 points, all significant.

So "is there an open-source model that replaces Jev?" was the wrong question. The models are not the variable — the labels are. A 310M encoder with 250 labels beats a decision API on one task and ties it on two; the same architecture with zero labels loses to everything.

I also paid for this lesson twice. Before this experiment I ran a proper evaluation of the two models most often named as Jev replacements — SemIf and Laya — on these same rows. Both failed to replace anything. Laya scored 29.2% on a 9-class task (chance 11.1%) with a median confidence of 0.029, i.e. it was visibly not deciding. Its zero-shot English number in its own documentation is 0.362, so this is not a Japanese-specific defect — it is a zero-shot defect. SemIf was closer but lost on all three tasks, two of them significantly.

Independent benchmarks showed the same thing. On a frozen-question benchmark, Laya's "hard" score was 34.1% while its model card implied a win over Jev, because the model had been fine-tuned on that benchmark's training split. Self-reported comparisons of fine-tuned models are not evidence about zero-shot behaviour.

Finding 2: training pays on topic tasks and does not on meaning tasks

Look at where the 12-point win came from and where it did not.

• livedoor (9-class topic, 1,174 chars): trained 88.8% vs Jev 76.8%. The labels are in the text — if you have a few hundred examples, "what topic is this" is a vocabulary problem, and a trained encoder learns the vocabulary. It also beat the char n-gram baseline by a hair (+0.4pt), which tells you the topic signal itself was already largely lexical.
• rakuten (2-class polarity, 138 chars): trained 92.8% vs Jev 94.4% — a statistical tie (p=0.50). Same for the local 26B MoE (p=0.45). Adding data does not fix this, because the problem is not "I don't know the vocabulary", it is "I need to judge sentiment".
• chabsa (3-class financial polarity, 92 chars): trained 75.2%, Jev 74.0%, local MoE 75.2%. A three-way tie. On this task no one is good — 74–75% is the ceiling for all three approaches, and that is the real finding. If you need better here, you need a third stage (a stronger model on the low-confidence tail), not a different single model.

The practical rule I now use: if the label is visible in the words, train something small; if the label is a judgement about the words, use a decision API; if both plateau around 75%, stop shopping for models and fix the data or add an escalation stage.

Finding 3: the trained encoder is dramatically cheaper to run

Latency, single-threaded, same machine:

That is 4–20× faster than the API, with zero marginal cost, no network dependency, and no data leaving the machine. The confidence score is a softmax you can calibrate yourself rather than a vendor's number.

This is the part that changes architecture. Once the local model is both faster and not worse, the API becomes the fallback rather than the primary path: trained encoder first, Jev only on the low-confidence tail, the novel-looking rows, and as a drift detector.

Because you will ask: can I train it on the GPU in the box?

I tried that too, on the same mini-PC (Ryzen 9 7940HS, Radeon 780M iGPU, gfx1103). Short answer: it works, and it is not worth switching to.

Getting PyTorch onto that iGPU is now possible through AMD's device-specific wheels, but there are three traps in a row:

• Installing with the AMD index as the only index makes pip fail to resolve ordinary dependencies, because you have just replaced PyPI.
• Adding PyPI back as an extra index lets PyPI's newer torch win the version comparison — you silently get a CUDA build and a pile of nvidia-* packages.
• The nightly index's newest torch requires a rocm-bootstrap version that the same index does not ship.

Once pinned properly, the iGPU is real: torch 2.8.0+rocm7.14.0a20260519, arch=gfx1103, fp16 matmul fine. The microbenchmark is impressive — 1.21 s/step against 16.41 s/step on CPU, 13.6×.

The real workload does not deliver that:

Two reasons the microbenchmark lies. Short sequences leave the GPU idle while the CPU is comparatively efficient at small matrices, and per-fold fixed costs (model load, tokenisation, one-at-a-time evaluation) can exceed the training itself. Only long-document training showed a real gain.

And it is not stable by default. Roughly every few minutes the driver resets the GPU — amdgpu: ring gfx_0.0.0 timeout → GPU reset(8) → then the process dies with HIP error: unspecified launch failure. The GPU edge temperature was 49°C, so this is a driver problem, not a thermal one. Serialising kernel dispatches and pinning the performance level plus running one fold per process with retries did get a full 5-fold run through cleanly, but fp16 also cost 2.8–4.4 points of accuracy (not significant at n=250, but negative on both tasks).

So: CPU for training, GPU for what already works — Vulkan inference. If you are training on your own private data, a documented 30 minutes per dataset on CPU is better than a 2.5× speedup that crashes.

Where the labels come from is the whole game

If trained beats zero-shot this decisively, the obvious move is to generate labels with Jev and train on those. That works, with one trap I want to be explicit about, because it is the reason I did not just ship the first version:

If Jev alone produced the labels, Jev's accuracy is the ceiling of the student. You are teaching a model to reproduce another model's mistakes. The fix is to spend the labels where they matter: run Jev over everything, then take the low-confidence rows and have a stronger model or a human relabel them. That is where the ceiling comes off, and it is also where the marginal cost is, because high-confidence rows can be accepted as-is.

At 250 labels the trained encoder is already at parity with Jev on the tasks Jev is strong at, and ahead where Jev is weak. Labels are the lever, and the second and third hundred labels are worth more than the first.

What I run now

Caveats

• The trained numbers are 5-fold CV over 250 rows, so 200 rows of training data per fold. That is a floor, not a ceiling — more labels should widen the topic-task margin.
• livedoor was truncated at 512 tokens, so the longest documents are not fully represented.
• The trained side reports raw argmax; I have not temperature-scaled it, so its confidence is not yet comparable to Jev's for thresholding.
• The char n-gram baseline is a single configuration. It may not be the strongest non-neural baseline available.
• The iGPU accuracy comparison is at n=250 per task, which is not enough to make a 3-point effect significant. "Not significant" is not "no effect."

Takeaway: Zero-shot and trained are different products, not better and worse versions of the same one. If you have no labels, a decision API is the only thing that works today. If you have a few hundred labels and the label is visible in the text, a 310M encoder will beat it, cost nothing to run, and answer in a tenth of a second. Measure your task's shape before you pick a winner — and be suspicious of any "open-source Jev replacement" benchmark where the model was trained on the benchmark.

The API used here: TypeSafe AI's Jev (System One, jev-latest) — text plus a list of questions in, a 0–1 score per question out, own confidence included, roughly 32k of context, $0.042 per 1M input tokens with output free. Both controlled and randomised question sets were frozen and committed before the runs, and every comparison above is on identical row ids.

If you have run the same shape of experiment on your own data, I want to know where a trained small model beat the API and where it did not — especially on tasks where both plateaued.

Related reading

• Jev: one judge call, or twelve dimension scores? — the same rows, attacking the other axis: decomposing one judgment into scored dimensions.
• Japanese embeddings, measured — the retrieval side of the same question.
• Self-hosted LLM economics — when running the model yourself actually pays.