World Model Benchmarks Need Receipts, Not Just Scores

HarnessEval-W landed on arXiv this week with a useful claim hidden inside a world-model benchmark: an eval should explain itself.

That sounds obvious until you look at how a lot of model evaluation works in practice. A benchmark gives you a scalar score. A leaderboard sorts the rows. A paper quotes the delta. Then people argue about whether the delta matters.

Then the model fails in a way the number cannot describe.

HarnessEval-W is aimed at interactive world models, the systems that generate or roll forward visual environments. That is a messy place to evaluate. You are not just asking whether a frame looks pretty. You are asking whether the state of the world stayed coherent after a camera move, whether an object persisted after leaving the screen, whether a physical action produced the right kind of response, and whether the whole rollout obeyed enough causality to be useful.

Humans can spot those failures quickly. A cup jumps across the table. A door opens the wrong way. A room changes layout after a pan. The hard part is making a benchmark catch that failure and leave behind enough evidence for someone else to audit it.

That is the useful part of HarnessEval-W.

The score is not the product

The paper describes HarnessEval-W as an "agentified evaluation pipeline." For each evaluation case, a planner reads the context, picks relevant skills, breaks the question into measurable subproblems, sends those to specialized sub-agents, then validates and aggregates the evidence into a final verdict.

In the project page's July 2026 snapshot, that means 330 evaluation cases, 11 specialized evaluation skills, 5,940 scored rollouts, and 18 models on the leaderboard. The authors report a 0.93 Spearman correlation with human Bradley-Terry ranking on intentional transitions, 71.7% pairwise accuracy against human choices on physical cases, and a repeat-evaluation envelope 4.9x narrower than WBench across three runs.

Those numbers are useful. The part worth stealing is the trace.

Every case becomes an evidence tree. The benchmark records what was tested, which visual grounding tools were used, which skills were activated or skipped, what each sub-agent concluded, and how the parent agent rolled that evidence into the final score.

A leaderboard row is usually too compressed to debug. If Model A beats Model B by 3 points, you still do not know whether it handled physics better, preserved offscreen state, rendered cleaner frames, or simply got lucky on easy cases. You also do not know whether the benchmark is grading the thing you care about.

A trace does not make the benchmark automatically right. It makes the benchmark inspectable.

That is the bar I want from agent evals.

World models make the eval problem hard to hide

Interactive world models are a good stress test because failure is often temporal.

A still image can look fine. The next ten seconds can betray that the model has no stable notion of the room. The first-person camera turns around, comes back, and the chair is now a plant. A ball rolls offscreen and apparently enters witness protection. A hand pushes an object and the scene reacts like a dream that forgot its own props.

A single perceptual metric does not catch that well. Even a good aggregate score can hide which kind of failure happened.

HarnessEval-W splits the evaluation into three axes: observation quality, transition correctness, and world persistence. Those break down into settings like render quality, physical observation, exploratory transitions, intentional transitions, physical transitions, drift resistance, revisit consistency, and offscreen evolution.

That taxonomy is useful because it stops the benchmark from pretending "world model quality" is one thing. A model can render beautifully and still fail object permanence. Another can preserve layout but botch physical responses. For anyone building on top of these systems, those are different risks.

The same pattern shows up outside video.

A coding agent can pass unit tests while making a migration impossible to review. A document agent can extract clean Markdown while losing table semantics. A browser agent can complete a happy-path checkout and still click the wrong destructive control when the UI shifts.

A useful eval does not say "agent good." It tells you which failure family you are buying.

Agentic evals need their own guardrails

There is a trap here, and it is worth naming.

Using agents to evaluate agents can turn into a very expensive mirror. If the evaluator is just another model generating a polished rationale after the fact, the trace becomes theater. You get a nicer PDF, not a more reliable measurement.

HarnessEval-W avoids some of that by forcing the evaluation through case-specific routing, discrete skills, visual evidence, validation, and saved case cards. The planner has to say why a skill applies. Sub-agents answer measurable questions. The parent validates evidence before aggregation. The project also compares judgments against human preferences and tests repeatability.

That is the part smaller teams should copy. Not the exact world-model machinery.

If an LLM is going to judge another LLM, the judge needs constraints. It should cite the artifact it inspected. It should answer narrow questions. It should save enough intermediate work that a human can check a disputed case. It should separate regression tests from capability tests. It should be boring enough that you can rerun it without turning the whole pipeline into a séance.

I am more interested in evals that leave receipts than evals that sound smart.

For agent workflows, a receipt can be simple. The diff the agent touched. The failing test it claimed to fix. The log line it used as evidence. The exact paragraph in a PDF where it found an answer. The permission boundary it hit and the question it asked before crossing it.

Without that, the final score is just vibes with decimals.

The practical version is smaller

Most teams do not need 11 evaluation skills and thousands of rollout scores. They need one harness that catches the failures their current agent actually makes.

For a coding agent, that might be a repo task where the obvious patch passes tests but breaks an invariant documented in an old issue. For a document agent, it might be a PDF with duplicate headings, footnotes, merged cells, and one section where the correct answer is "not enough information." For an ops agent, it might be a runbook where the safe path requires stopping before a risky command and asking for approval.

Scale is optional. Shape is not.

Break the job into questions. Attach evidence to each answer. Save the trace. Make the final verdict boring to audit.

That is also how you notice when an eval ages out. If the trace never shows disagreement, never catches a boundary case, and never teaches you why a model failed, the benchmark has probably become a regression test. That is still useful, but it is no longer telling you where capability moved.

HarnessEval-W is framed around visual world models, but the broader lesson is closer to everyday agent engineering. Once agents touch real workflows, benchmark scores are too thin by themselves. You need to know what the evaluator looked at, what it ignored, and why it believed the answer.

A score tells you who won the row. The receipt tells you whether the row is worth trusting.