We gave our LLM trading bot an auditor that never sees the chart

Every hour, on the hour, a vision model looks at three forex charts and decides whether our bot should buy, sell, or sit out. It has been doing this with real money since May. As of last Saturday, something finally grades its reasoning.

This post is about the grader.

The bot, briefly

It's a Python script called gemini_snap_flash. Once an hour it sends a one-minute candlestick chart plus daily and weekly context images to Gemini, gets back UP or DOWN and a one-line thesis, and trades EURUSD, GBPUSD, or USDJPY through MT5 on funded prop accounts. One position per pair, fixed stop and take-profit, and hard rails around it: 2% daily loss cap, 0.6% daily profit cap, and an external watchdog that kills the process after four losses in a row.

The rails matter. The first $10k account blew through its max drawdown on September 9 and is gone. The fleet is two $100k accounts now, run as separate instances with separate state.

Here's the part most AI-trading posts skip. The bot already logs a lot. signal_log.csv has every decision, executed or skipped, with the model's reasoning attached. trade_analysis.csv has every close with MFE/MAE in pips and R-multiples. What nothing had was a check on the decision itself: does the thesis actually argue for the direction the model picked? When the account died, I could read every trade and every excuse, and none of it told me whether the reasoning was any good.

What JEV is

JEV is TypeSafe's System-One model (jev-latest), served at api.typesafe.ai/v1/systemone. It's text-only. No vision, no market opinions. You POST it a "state" string plus a set of questions, each with instructions and pass/fail criteria, and it returns structured verdicts:

{
  "state": "<the thing being judged>",
  "model": "jev-latest",
  "questions": {
    "shadow_direction_consistency": {
      "type": "noul",
      "instructions": "Based only on the provided state (a recorded hourly vision decision for a live FX scalper), is the recorded directional call a clear, self-consistent directional commitment coherent with its own thesis?",
      "criteria": {
        "true": "The recorded direction is explicit (UP or DOWN) and the recorded thesis is coherent with and supports it.",
        "false": "The direction is ambiguous, contradicts the thesis, or no clear directional commitment is recorded."
      }
    }
  }
}

The shape isn't an accident. A noul question is a pass/fail check the model answers one way against criteria you wrote — not an open-ended essay. That's what makes the verdict usable as a watchdog: it stays binary and countable, so you can line thousands of them up against the trades they judged and watch the incoherence rate move. An essay grader couldn't do that; it would drown you in prose and give you nothing to plot.

So it's a proofreader, not a fortune teller. It can't tell you whether UP is right about EURUSD. It can tell you when UP arrives with a thesis that quietly argues DOWN.

The shadow harness

The new file is shadow.py, about 430 lines, plus tests. The design has one hard rule: the trading path must not know it exists unless we say so.

Everything is gated behind SHADOW_MODE=1. When that's unset, the harness is a complete no-op; the only cost on the hot path is a single os.getenv() call, and the tests assert that zero network calls happen. When it's on, for every hourly vision decision the harness:

1. Dumps the exact chart images that went to the vision model into shadow_out/<instance>/<timestamp>/, and records a sha256 of the prompt text for later lookup.
2. Runs exactly one JEV call over a text rendering of those same inputs, hard-timeboxed at 20 seconds on a daemon thread. The state is capped at 30k characters to stay inside the model's context.
3. Appends one JSON line per decision to shadow_log.jsonl, including the JEV verdict and its latency.

A little later, an attach_outcome() call patches the most recent unfinalized line for that pair with what actually happened: EXECUTED, FAILED, or ORDER_FAILED, plus the ticket. It only touches lines written in the last 15 minutes, so a slow hourly cycle can't corrupt an older decision's record.

Each instance writes to its own log, so the fleet's accounts never mix.

The switch is currently off

Worth saying plainly: SHADOW_MODE is not set in any launch config yet, so zero shadow calls have happened in production. The dashboard probe ledger shows nine calls, all from us poking at the API while building the plugin, and the Shadow Harness tab reads NOT ENABLED until the first row lands. The harness is new and the tests are new, and we want a few market-open cycles of confidence before turning on a collector that sits in the decision path.

What we expect the data to answer

Three questions, in order of interest.

First, the big one: does coherence predict money? Every verdict lands in the same record as the real action and, downstream, the real P&L. If incoherent trades lose measurably more often than coherent ones, JEV graduates from observer to entry filter, and we skip or downsize flagged decisions. If it doesn't, we drop it. That's the point of measuring instead of believing.

How a verdict gets scored is fixed before the first data point. The pass/fail bar is pre-registered in code — minimum sample sizes, a minimum counterfactual edge, a ceiling on JEV errors — and the backend and the dashboard banner read the same constants, so the goalposts can't move after the data arrives. That same tab is where we watch it: it shows one of five states — NOT ENABLED, INSUFFICIENT, SHIP CANDIDATE, KILL, HOLD — next to the histogram of disagreement edges and a per-day series. The edge itself is computed counterfactually: on rows where JEV withholds endorsement, we flip the sign of what the vision-side trade actually realized. That's a mirror approximation, not a P&L replay, and it's labelled as one; orders that never resolved are counted separately, never scored as zero. And if JEV's disagreements turn out to correlate with winners, the shadow signal dies first.

Second, drift detection. The bot retries Gemini up to six times with exponential backoff, and we've swapped models before (3.5-flash to 3.6-flash in July). Degraded or fallback responses usually get sloppier before they get unprofitable. A rise in inconsistent verdicts is a cheap alarm — a shadow call is text-only, input-only pricing, a fraction of a cent per decision, and every record carries the JEV latency and error status, so degradation shows up in the record itself before the drawdown alert does.

Third, model comparison. Since the harness records each prompt's hash and the decision but stays agnostic about which model produced them, the same record can score an OpenRouter alternative against Gemini on identical charts. Evidence beats vibes when picking a model.

What it can't do

The limits are real and worth naming. JEV sees the text of the decision, never the chart pixels, so it audits self-consistency only. It will happily bless a confident, well-written, wrong call. It evaluates after the fact, so today it can't stop a trade, which is deliberate: any gating step is a later change with its own risk to review. And it's one question right now. Coverage is only as good as the question set, and one question is a start, not a panel. The API shape makes a panel cheap, though: several independent pass/fail checks can ride in a single request and get combined in code, and our dashboard probe already does that. Adding questions to the harness is a config change, not a rewrite.

The whole thing is built to answer one question, and the answer is allowed to be no. If coherent trades don't make more money than incoherent ones, JEV gets dropped and we've lost a few cents a decision and a couple of weeks. If they do, the auditor stops being a proofreader and becomes a filter. Either way the data decides, and the bar was written before the first verdict landed. That's the part I'm actually proud of — not the harness, the willingness to let it be wrong.