Human, agents, code, judge: adding Jev without replacing peer review

I use Claude and GPT to review each other's work. Adding Jev did not replace that arrangement. It gave it another instrument.

The useful distinction is between an agent reviewing a whole change and a model answering a narrow question about supplied evidence. I want both. I also need to check the scripts connecting them.

What prompted the experiment

One of my projects has a growing documentation library. After a milestone, the current status changed, but older descriptions remained scattered across the docs.

Two cleanup passes had already happened on the Claude side. A subsequent semantic scan produced candidates that the agent checked against the repository. Nine stale statements were confirmed and corrected.

One was especially revealing: an old “corrected” marker exempted a sentence from the existing lexical guard rules. The marker was still there; the statement had become outdated again.

This was not Claude and GPT both missing the same nine statements. Those two passes came from the same agent/model family. Nor does this establish that a judge is a better general reviewer. It shows why a separately framed check was useful in this incident.

Four layers, with review across them

Claude and GPT occupy the same layer. Neither is above the other. Either can review the other's work — including changes to the checking tools.

This is a division of authority, not a ranking of intelligence. An agent's operational responsibility means assigned work, traceable actions, and correction when challenged. It does not transfer my human accountability to a model.

The judge has no permission to edit, publish, or approve a release in this setup. That follows from the tools and permissions I give it, not from whether it can say “I was wrong.” A speaking model would not earn those permissions just by sounding responsible.

What “a model without words” actually means

Jev accepts state and typed questions rather than a request for a written review. Its question types cover a yes/no judgment, a choice among options, and a position on a defined scale. It returns structured results. TypeSafe's primitives documentation describes the interface.

That makes the output convenient for code. It does not make it inherently truthful.

A wrong answer can be a number. A narrow output format cannot repair missing evidence, a misleading question, or options that omit the right answer. The vendor also distinguishes calibration across predictions from a guarantee about an individual answer. System One documentation

My rule is simple: a model's confidence is a signal to investigate, not permission to act. Thresholds need testing on the task at hand. A low-confidence result may justify abstaining; a high-confidence result can still be wrong.

The check I actually run

For document freshness, the workflow is:

1. Establish a dated fact sheet from the project's current records.

2. Select current documents, leaving intentionally historical records distinguishable.

3. Extract sentences and list items with enough context to interpret them.

4. Ask whether a statement conflicts with those supplied facts.

5. Have an agent inspect candidates against the source before changing anything.

This describes scans I have run. It is not an automatic check after every commit, and it does not cover every file or every possible defect.

Some false alarms came from my own framing: a fact written too broadly made valid historical statements look stale. Changing question wording also changed which statements were flagged. That means the fact sheet and question deserve review alongside the document.

Low-priority results are not certificates of correctness. If I inspect only flagged statements, I can discover mistakes but cannot measure everything the scan missed. Known counterexamples and samples of unflagged material belong in any stronger evaluation.

The same structure can check a report against a command result: “supported,” “contradicted,” or “not established by this evidence.” The third option matters. Missing support is not automatically a contradiction.

What the GPT-side review added

The independent GPT-side pass tested a small reading task in another project, using both claims present in an excerpt and opposing claims the excerpt did not support. It kept a distinction I want throughout the system: recognizing what a passage says is not verifying whether the passage is true.

That review also examined the toolkit itself. It found gaps between the documented data policy and its implementation. Passing tests had not established every boundary we were describing.

This is a different contribution from finding stale prose. A judge inspecting supplied sentences cannot be assumed to notice a problem in the client that prepares its requests. Another agent can inspect that client, challenge its assumptions, and add a concrete test.

The new layer therefore strengthened the case for peer review. It did not remove it.

Keeping private work local

My policy separates content permitted for a hosted service from local-only material. Secrets and customer records are excluded. The local path uses a server on the machine's loopback interface.

But writing that policy is not the same as enforcing it. Request preparation, destinations, redirects, and credential handling need code checks. Content selection still needs judgment: a label saying “allowed” does not prove that the bytes belong in that category.

I inspect prepared requests and treat the toolkit as a work in progress, not a verified end-to-end security boundary. Nothing about a judge's answer makes an unauthorized transfer acceptable.

The local experiments were instructive — and limited

I also tried local models on an M1 Mac with 8 GB of memory: Qwen2.5 1.5B and 3B in 4-bit form, then Bonsai 27B 1-bit.

These are small exploratory tests of particular prompts and configurations, not a general model ranking or a held-out benchmark.

In the single-pass setup, the small Qwen models did not give useful separation for the sentence-freshness task. Allowing the 3B model a short reasoning step improved that task, but did not establish a dependable general claim checker.

Bonsai matched 12 of 14 labels on a small synthetic claim/evidence set. In a later mixed set of ten project-status claims, including planted false statements and an unrelated claim, its top-choice labels matched eight. One apparently correct contradiction had weak confidence: label agreement was not the same as a reliable automatic alert.

It also missed a deliberately stale index entry in a longer-context test. A “nothing found” result was not reassuring once a known defect had escaped it. I kept that task manual.

These local models can generate language; the adapter restricts what the caller receives. That is not the same architecture as Jev, and the similar interface does not establish equivalent calibration or behavior.

The Bonsai run used the project's custom llama.cpp fork. Its model card documents the specialized weight format and kernels. Fitting the weights was only part of the resource question: the machine showed substantial swap during these runs. That was a system-wide observation, not a measurement attributing all swap to one process.

For me, the local option is a bounded experiment, not an always-running background service.

What I am keeping

The useful output of the experiment was not an infallible final reviewer. It was a clearer allocation of work.

Agents can challenge the task, the fact sheet, the code, and each other. Deterministic checks establish specific properties. A judge helps identify statements worth another look. I keep the human decisions I have not delegated.

All of those layers can be wrong in different ways — and shared assumptions can make them wrong together. I am trying to make correction easier, not make another model's answer the end of the conversation.