Key takeaways
- Prove the lever can be applied before you A/B it, and make the engine state its own resolved config — a harness that prints what it set is not evidence of what the process read.
- Prove the grader in both directions: a known-good build must score high, the same tree twice must produce an identical vector, and each injected defect must break only its own checks.
- Detect injected defects by delta against the known-good's own vector, never by membership in a fail list.
- Positive-control every zero. A count of nothing is only evidence if you first proved the counter can see something.
- When the score is right but the report misleads, fix the report. Marking blocked checks 'unscored' pays a broken app for failing early.
The lie is almost never the model
I have been building an execution-graded benchmark for agentic coding inside goose Local Edition, our fork of Block's goose. It boots the application an agent produced, drives it over HTTP and in a headless browser, kills it, restarts it, and prices what a user would actually experience. That article covers what it is. This one covers what it took, which is a different and much less flattering story.
Here is the summary I did not want to write. Every number this benchmark has ever published was wrong at least once. Not "refined later" but wrong, and caught, and in the worst case caught by a human looking at a screenshot rather than by any gate I built. The scorer itself is the least interesting thing I can hand you. What transfers is the set of disciplines that caught each lie, because every one of them started as a number that looked completely reasonable.
One thing to declare before anything else, because it changes how you should read the rest. The system under test and the system doing the testing are the same fork. goose builds the app; the same repository holds the scorer that grades it; the swarm's engine and the scorer share the browser probe. That is a conflict of interest and it is exactly why the controls in the section on proving the grader exist. A grader written by the people whose runs it grades has to be provable in both directions or it is worth nothing. Where the engine and the scorer share code I will say so, because one measurement bug turned out to be simultaneously a scoring bug and a repair bug for precisely that reason.
Everything I quote below is in the public repo at leanzero-srl/goose-local-edition, under evals/swarm-bench/. I re-read all of it against the remote branch while writing this, because my own checkout was 1,532 commits behind.
Your first adversary is the config plumbing
I want to open on the least expected part, because it is the most transferable.
Before you can A/B a feature flag you have to prove the flag can be applied at all. For a long stretch of this project it could not be, and nothing said so. The engine has a family of GOOSE_SWARM_* gates, and the campaign that was supposed to measure them ran with a harness that printed lines like arm env: GOOSE_SWARM_X=1 for a week. The harness was telling the truth about what it had set. It was saying nothing about what the process read.
The doc comment on swarm_gate_cfg_bundle in swarm.rs states the diagnosis flatly:
PROVEN: the app is launched with
open -n Goose.app, andopenhands the spawn to LaunchServices, which gives the app its own environment —env FOO=1 open -n Appsets FOO foropen, which then exits. So everyGOOSE_SWARM_*var intended for the desktop has always been discarded, and the whole lever campaign silently ran with every env-gated lever OFF. config.yaml is the only channel that reaches the engine.
I tried to reproduce that mechanism on this machine today and I could not. I built a minimal ad-hoc-signed app bundle whose only job is to write getenv("LZ_LEVER") to a file, registered it with LaunchServices, and ran the launch four times on macOS 26.6.1:
control (no env set) LZ_LEVER=<unset>
env LZ_LEVER=run1 open -n App LZ_LEVER=run1
env LZ_LEVER=run2 open -n App LZ_LEVER=run2
env LZ_LEVER=run3 open -n App LZ_LEVER=run3
Four for four, with a clean negative control. It reproduced on a second clean build of the same bundle, and again with the bundle installed in /Applications rather than a home directory, so the install location is not the difference. On this OS version, with that bundle, open -n delivers the environment. What does reproduce, deterministically, is the variant without -n: with an instance already running, open launches no new process at all, so the freshly-set variable reaches nothing and the running instance keeps whatever environment it was born with.
env LZ_LEVER=first open -n App -> pid=12674 LZ_LEVER=first
env LZ_LEVER=second open -n App -> pid=12680 LZ_LEVER=second
env LZ_LEVER=third open App -> (no new process; nothing read anything)
I could not finish the third variant, which is the one I most wanted: an app carrying the com.apple.security.app-sandbox entitlement. Ad-hoc-signed and without a provisioning profile, it produced no container and no output, so whether the sandbox scrubs the environment is untested here and I am not going to guess at it. The shipping app is a signed, notarized Electron build on a macOS version I cannot roll back to, and I am not going to reason backwards from a bundle I made in a minute to one I did not.
So the honest statement is narrower than the one in the source comment, and it makes the lesson stronger rather than weaker. Something took those levers out. The recorded mechanism does not reproduce on this machine today. Even the post-mortem of an instrument failure was itself an inference that nobody had re-run, which is precisely the class of mistake the whole exercise exists to catch.
And the remedy is unchanged, because the remedy never depended on the diagnosis. Commit 50fa4654a made the engine state its own configuration in-band. It emits a levers_resolved event whose values are computed by calling the same expressions the engine branches on, so the record cannot drift from the behaviour without the behaviour changing:
let mut levers_event = serde_json::json!({
"event": "levers_resolved",
"version": option_env!("GOOSE_BUILD_VERSION").unwrap_or("dev"),
"build_sha": option_env!("GOOSE_BUILD_SHA").unwrap_or("dev"),
"crate_version": env!("CARGO_PKG_VERSION"),
"levers": { /* every gate, resolved */ },
});
The version field is there for a reason worth stealing. CARGO_PKG_VERSION alone is useless: the workspace crate version is 1.41.0 and has not moved in 54 desktop releases, so every run from every build ever emits the same string. A number the engine did not emit is not evidence.
Two more details from the same file carry the whole article in miniature. First, the precedence logic was split into two pure functions, resolve_gate and resolve_gate_cfg, because — in the comment's own words — "the test for this used to re-type the unwrap_or_else chain against literals, so it asserted on a COPY of the logic and would have passed even if the real function drifted away from it." A test that reimplements the thing it tests is decoration.
Second, and this is the sentence I would put on a wall:
levers_resolvedis a hand-maintained list, so a new gate is invisible until someone adds it — and "absent from the event" reads EXACTLY like "resolved to null".
Absence and zero must never be spelled the same way. Every remaining section is a variation on that one rule.
The graded task is a product spec, not a puzzle
The task is spec-build-v2.md, 125 lines, and the entrant builds a payments sync tool called vendorsync. Five files by path. Exact MeridianClient and Store signatures. A four-row HTTP table. limit defaults to 25 and caps at 100. Rows carry exactly id, amount_minor, currency, created_at, status. Unknown path returns 404 {"error": "not_found"}; a bad limit or offset returns 400 {"error": "bad_request"}.
The backend half is ordinary. The frontend half is where it stops being a demo. Three separately-owned files. Dates rendered in the user's locale, with the spec saying outright that "A raw ISO-8601 string with an offset (2026-03-01T14:00:00+00:00) must never appear in the rendered page". Statuses that "must differ in computed color, not only in their text". Prev and Next plus a showing X–Y of TOTAL readout. A status filter that "must actually change the rows shown". No horizontal scroll at 375 px. Distinct loading, empty and error states, each showing text a user can act on.
The spec even carries a design rule set, printed verbatim in the brief the agent receives: never decorate cards or rows with a left accent line or rail, no default <select> for the filter, no alert() or confirm(). Three budgets are in there too — page interactive under 2 s, /api/payments?limit=25 under 150 ms at p95, a full sync of the 247-row fixture inside 60 s.
The structural point matters more than any individual rule. The scorer never invents a requirement. Every check quotes a sentence the builder was given, which is what makes a deduction defensible instead of opinionated. modules_present wants precisely the five paths the spec names, and nothing else.
A score is a weighted mean of means
Walk the arithmetic, because "47 out of 60" hides everything that matters.
TIER_WEIGHT = {"A": 0.25, "B": 0.30, "C": 0.25, "D": 0.20}
PRODUCT_WEIGHT = {"J": 0.15, "V": 0.10, "P": 0.05}
PRODUCT_CORE = 0.60
score = PRODUCT_CORE * core_weighted + product_weighted + 0.10 * hard_mean
Six checks form a hard block — request_efficiency, second_sync_cost, client_create_replay, client_idempotency_key, update_propagation, restart_persistence — and they are excluded from their home tiers' means so their difficulty cannot be diluted by easier tier-mates. I ran the composition to check the claimed consequence rather than trusting the comment:
everything else perfect, hard_mean=0.0 -> score 0.9000
everything else perfect, hard_mean=0.5 -> score 0.9500
everything else perfect, hard_mean=1.0 -> score 1.0000
A core-perfect build caps at exactly 0.90, so 0.996-by-saturation is structurally impossible. The final tenth is earnable only from checks that stay hard by construction.
Now the consequence you can feel. Counting the registry myself against the remote branch: the sb-5 scorer registers 60 checks across seven tiers, A 6, B 16, C 15, D 9, J 5, P 3, V 6. Sixteen checks in tier B makes each one cheap. A B check is worth 0.60 × 0.30 / 16 = 0.01125 of the final score. A journey check is worth 0.15 / 5 = 0.030, which is 2.7 times more. Nobody chose that ratio; it fell out of how many checks happened to land in each tier.
Everything is graded in [0,1], never binary, and the module docstring gives the reason: "A binary sheet cannot express that — it collapses 'returned 3 of 47 payments' and 'returned 46 of 47' into the same MISS." So chronological_order_full scores the fraction of adjacent pairs in true instant order across the whole collection. update_propagation returns min(update_seen / update_changed, 1.0) over the 25 rows the vendor mutates. restart_persistence returns surviving rows over rows before the SIGKILL.
And the design rule that reframes the genre, straight from the same docstring:
100 is not meant to be reachable. A perfect score means the task has stopped measuring … If something ever does reach 100, that is the signal to deepen the task, not to celebrate.
Four instruments, and one of them lives in a subprocess
gather() is the part you can copy the shape of. It boots python -m vendorsync on a free port, polls /api/health for 25 seconds, and then drives the app through a fixed battery: two syncs, each bracketed by a phase marker; a paging matrix; a validation matrix of bad limits, bad offsets and unknown paths, six cells of it (the comment above it still says seven); three raw wire-byte pages at offsets 0, 100 and 200 so that the float sentinel in row_integrity sees bytes rather than Python's post-parse view; two overlapping POST /api/sync calls in threads; a six-path JSON sample across success and error paths; a vendor-side mutation of 25 statuses followed by a third sync; timing probes; four headless-browser scenarios covering load, sync, error and empty; and finally os.killpg(os.getpgid(proc.pid), SIGKILL) and a respawn on the same --db, counting rows back through the app's own API. A comment forbids opening the SQLite file directly after the kill, because a read-only URI open can miss WAL-committed rows.
A fifth instrument is deliberately separate. probes/client_probe.py writes a driver script and runs it as a subprocess from the tree root with a 180-second cap, emitting one JSON object on stdout. It imports the built MeridianClient and calls fetch_all_payments(), total_count() and create_payment() against its own fresh vendor instance on its own port and its own trace, "so it can never contaminate the main exercise's trace or trap state". Each step is guarded independently, so one failing method loses its own checks and nothing else. That guard has a name in the ledger: the 0/44-collapse lesson.
Then there are the static checks, and they should be named as the weak ones rather than buried. interfaces_declared parses with ast. store_atomic_upsert pulls the insert statement out of store.py with re.search(r"INSERT[^;\"']*?INTO\s+payments.*?(?=\"\"\"|;|\\Z)") and inspects that. client_timeouts is a single re.search(r"timeout\s*=") over meridian.py. Grep-grade checks are grep-grade, and further down I will show you exactly what that cost.
The general shape here is the one I wrote about in swarm-gym for goose: grade by running the code, not by reading it. This is that idea taken about as far as it goes.
The vendor contract is an audit, not a unit-test suite
The C tier is graded from the request trace of a mock vendor, and the header of vendor_service.py sets the fairness bar:
you cannot fake having sent
If-None-Match, and you cannot fake having waited forRetry-After… Nothing is a trick: a careful engineer who reads the docs gets every one of them right, which is the fairness bar the benchmark holds itself to.
Six documented behaviours, all keyed to the client's Nth list request within the graded phase rather than to a page number. The 2nd list request answers 429 with Retry-After in seconds. The 3rd, the cursor-bearing retry, answers 410 cursor_expired, with the documented recovery being a restart from the first page. The first request after that answers 429 with Retry-After as an RFC 7231 HTTP-date. There is a deliberately short page at offset 25 that is explicitly not the end of the collection. Per-page ETags answer 304, and their generation counter bumps on any mutation, because "hashing only offset+limit taught clients that ETags lie". And POST /v1/payments answers 409 with the same payment_id when an idempotency key is replayed.
request_efficiency grades against an optimum of 7 requests, and the comment insists that number was measured rather than guessed. I did not want to take that on trust, so I stood the mock up on this machine, called begin_exercise_phase() exactly as the scorer does, and walked the chain with a client that pages at the documented maximum and honours what it is told:
# offset status error Retry-After
1 0 200
2 100 429 rate_limited 2
3 100 410 cursor_expired
4 0 429 rate_limited Wed, 26 Aug 2026 05:14:39 GMT
5 0 200
6 100 200
7 200 200
rows fetched : 247 unique ids : 247 restarts : 1 REQUESTS : 7
Seven, with all 247 rows and no duplicates. The constant is real, and the trap chain is unavoidable for a correct client, which is why charging for it would be a deduction no implementation could escape.
Then I ran two more clients through the same mock, and the second one is the interesting result:
correct client, default limit=25 requests=15 rows=247 raw ratio 0.467
naive: stops when a page is short requests= 6 rows= 30 raw ratio 1.167
The naive client stops at the short page at offset 25, walks away with 30 of 247 payments, issues fewer requests than the optimum, and its raw efficiency ratio comes out above 1.0. The grade helper clamps to [0,1], so it scores a clean 1.0 for efficiency having fetched 12% of the data. That is a live vacuous pass, and I reproduced it from first principles in about a minute. It is not news to the project — the sb-6 data case records the same defect in the corpus, with 13 of 31 serious builds scoring 1.0 on 3 to 5 requests, of which 5 had synced zero rows — but reproducing it independently is the difference between believing a document and knowing a thing.
Each check names the harm it stands for. client_idempotency_key: "without the key a retry creates a second real payment — a double charge." The doctrine is in vendor_trace.py: "if the harm cannot be named, the check does not belong here."
One piece of craft that is easy to miss. mark_phase() and begin_exercise_phase() reset the one-shot traps and stamp the trace, so grading covers only what the delivered client did. Without it, the agent's own development testing is indistinguishable from the finished module. The motivating measurement is in the docstring: "Opus's one-shot 429 was consumed at trace seq 3 while the graded run began at seq 38, so the retry check graded throwaway scratch code."
Prove the grader before you believe a score
controls.py names three properties and then names the one nobody tests:
HIGH a known-good build scores high
LOW a deliberately broken build scores low
ISOLATION each injected defect fails ONLY its own check
Isolation is the one that matters most and is almost never checked. A grader with a shared precondition collapses a mostly-correct build to zero — measured twice in this project, once when a missing subcommand took a 43/45 build to 0/44, and once when a stale trace field scored a correct paginator 0%. Both looked like devastating model failures and were neither.
Thirteen defects are injected into a copy of a real known-good tree rather than a hand-written strawman. Delete web/. Rename total_count, chosen precisely because nothing calls it at runtime. Turn the API's default 25 into 7. Replace the atomic upsert with INSERT OR REPLACE and strip ON CONFLICT(id). Change == 409 to == 499. Make the store open sqlite3.connect(str(self.path) + str(os.getpid())) so persistence dies at the SIGKILL and nowhere else. Turn status=excluded.status into status=payments.status. Plus six frontend defects, including returning the raw ISO string from the date formatter, blanking a badge's class, setting the page size to 1000, hiding the filter group with an inline style, an injected early return; in the sync handler, and a deferred throw.
The move worth stealing is how detection works:
newly = {c["check"] for c in got["checks"]
if c["score"] < gv.get(c["check"], 1.0) - 1e-9}
Delta-based, not membership-based. A graded check that gets worse under a defect was invisible while the known-good already scored it below 1.0, and update_ignored went "undetected" exactly that way. Newly broken means the score dropped against the known-good's own vector.
Determinism is a real third control. The same tree is scored twice, after unlinking the -wal and -shm sidecars as well as the database, and any per-check disagreement is drift. The run ends with trusted = ok_high and ok_det and not failures and prints either GRADER TRUSTED or GRADER NOT TRUSTED. Be honest about what main() actually runs: HIGH, determinism and ISOLATION. There is no separate LOW pass.
And the controls themselves were wrong twice, which the file records rather than hides. The first version of the interface defect renamed last_sync, which is called on the health path; the server crashed and 24 checks cascaded. Deleting ON CONFLICT outright broke the SQL so nothing persisted at all. Both were bad controls, not bad graders, and knowing the difference is most of the skill.
Rendered means seen
This is the section the article exists for.
Commit c7aac3af70d413de6c2921c81ef592685734d68f, Tuesday 18 August 2026, 10:53:
Mihai read it straight off the screenshots. The app loads all 247 payments, appends its rows, throws on an undefined identifier in the same render pass, and paints the error state over a
display:nonetable. Every row counter inproduct_probe.mjsread the raw DOM —pageAnalyzeLoad,pageViewSnapshot,pageFirstDataMs— so hidden rows scored as rendered, inflating J/P and the frontend slice of B, flipping "excellent" on, and (the deeper harm) blinding the ENGINE's render gate, which shares this probe: no repair round was ever pointed at the frontend because the instrument said the frontend worked.
Three failures wearing one bug, and they are worth separating. A scoring bug, because the score inflated. A repair bug, because the engine shares the probe, so no fix round was ever aimed at the frontend. And a reporting bug, because the word "excellent" flipped on.
The fix is four lines of principle:
const domRows = dataRows();
const rows = domRows.filter(visible);
// the raw DOM count stays as a diagnostic so the gap itself is visible
const renderedRowCount = rows.length;
const domRowCount = domRows.length;
A row counts only if the browser would paint it, and the raw count survives as a diagnostic so the gap between the two is itself measurable. SCORER_VERSION went sb-5.2 to sb-5.3, the shipped tree was re-scored under the fixed probe at 0.8911, excellent false, V 0.55 and J 0.80, and the commit closes with the line that decided the site's editorial policy:
The 0.9528 row is not publishable and will not be published.
Two things deserve saying plainly. It was caught by a human looking at a screenshot, not by any automated gate I had built. And the commit already answers the obvious objection about the repair rounds: they "worked precisely the findings the gates named … and the one defect a human sees was on no list." The sb-5.3 fleet run card is one of the runs posted under the fixed probe. It carries four screenshots of the built application, labelled first render before repairs, final render, after sync, and mobile at 375 px, sitting on the same page as the per-check rows and their detail strings. That arrangement is the one that would have caught this on day one, and it is why it exists.
Everything else that was lying
Each of these is a different shape of lie, and not one of them looks like a lie while you are looking at it.
A zero that meant broken. Commit 5bb5640a0: "Of 133 rows, 104 ARE PHANTOM — 78%", each carrying a wall_secs of 0.1 to 0.2 seconds. Every lever arm was 100% phantom; only baseline had real runs, 26 of its 40. The instrument was fine and the wiring was not: each phantom carried harness_ok: false and the void flag was never set from it. The near-miss is the teaching moment, in the commit's own words: pooling those rows "would have scored the lever at 0.0000 and produced a crushing, entirely fabricated verdict against THE ONE LEVER BUILT TO FIX THE DEFECT I HAD JUST FOUND."
And the hero of that commit is not the analysis. The fixer script written to repair the corpus was itself blind. It globbed result.json when the file is nodeloop-result.json, opened zero files, and reported "0 rows fixed", which reads exactly like "nothing was wrong". "THE POSITIVE CONTROL — count the files opened — IS THE ONLY REASON THIS WAS FOUND." If you take one line from this article, take that one. It is the same failure I wrote up in pytest piped to head exits 0, where a test command that ran nothing at all returned a clean exit status. A count of nothing is only evidence once you have proved the counter can see something.
That commit is also a small candour lesson in itself: its subject line says 105 phantom rows and its body says 104 of 133. I am quoting the body, and the disagreement is worth leaving visible rather than averaging away.
The correction was also wrong. Commit 6aaa0a517 quotes its own headline back at itself — "THE ENTIRE LEVER CAMPAIGN HAS PRODUCED ZERO DATA … THAT IS WRONG" — after recovering five real runs from loop.log and finding that "25 OF 53 REAL RUNS ON DISK — 47% — ARE MISSING FROM THE CORPUS ENTIRELY". The cause is mundane and vicious: cell directories are reused, so a later phantom overwrote an earlier genuine result file. The corpus was a survivorship snapshot, and a prior finding was declared wrong the same way, having compared complete logs to result rows while reading the survivorship snapshot on both sides. No published number moved, and that was proved rather than asserted: 1-node n=11 mean 0.7327 against 3-node n=15 mean 0.7280, byte-identical before and after.
Zero time. Commit 5ed189bcf: all five judge-terminated emit sites hard-coded elapsed_ms: 0, while the elapsed time sat in scope one screen above. per_device.busy_ms is the engine's own answer to how busy each node was, which is the exact question the node-scaling goal turns on, and it silently dropped the commonest restart in the engine. An 80-minute task was recorded as taking no time at all on three of its five attempts, deleting 51.7 minutes from a 123-minute run.
The same commit kills a hypothesis with its own calibration, and I like it more than the fix. Every acted over_reading kill across all seven cells had landed on a worker whose deliverable is a test file: 15 test tasks and one non-test that happens to own tests/test_integration.py, so 16 of 16. The mechanism looked legible and the threshold was about to be widened for test tasks. Then the calibration: successful test tasks use a median of 5 tool calls and a maximum of 13, against a 16-call trip, so "the correct reading of 16-of-16 is the opposite of mine" — test tasks are the population that thrashes. No threshold change, in either direction.
Two populations in one column. Commit af9803882: 11 of 79 completed tasks, 14%, were progress-watchdog salvages of stalled tasks and were indistinguishable from clean successes in the log, so "every score and occupancy figure this campaign has mixed two populations without saying so".
The machine, not the artifact. Commit 4474b46a8: serial re-scores on a quiet machine, each tree alone and at its own advertised vendor port, moved Opus 0.7445 to 0.8281, Sonnet 0.7314 to 0.7666, Haiku 0.4518 to 0.4597, and Luna 0.7887 to 0.7887. The discriminating move is the per-check diff rather than the totals: "NONE of the moved checks are the three probe fixes. All of them are vendor-interaction timing." The afternoon trio had been scored in parallel, and webhook delivery races and stall-trap windows under that load read as application defects. Policy from there: scoring is serial, always. Separately, in commit 0af92c73a, a leftover warm database measured the hand-written golden reference at 1.00 and then 0.7995, on nothing but a stale file.
Two bugs wearing one symptom. Commit 511409d66 reproduced a 0-of-247 sync offline in seconds by importing each cell's own vendorsync.meridian against a real vendor fixture on a private port. One app read data.get("payments", []) when the vendor sends {"data": [...]}, and "the .get default swallows it, the page loop sees an empty list, and the sync reports success having stored nothing". Another read the right key and mishandled the cursor. The message then refuses the tidy conclusion it could have drawn from a four-of-four separation on one JSON key, writing instead: Explicitly NOT "one key explains the build".
The structural sentence in that commit is the deepest thing in the whole record, and it is about your app as much as mine. An app returning well-formed empty results satisfies the entire pipeline. It compiles, it runs, it returns 200s, its own tests pass because they mock the vendor with the assumed shape, and the contract gate called the repeat sync idempotent and counted it verified. Nothing ever exercised the app against the real vendor.
Pre-register the falsifier, and the marker
A falsifier here is a prediction written into a file before the evidence lands, with the exact observable that would refute it. It also carries a marker: the string literal whose presence in the running binary is what makes the prediction settleable at all. From PREDICTIONS:
A prediction can only be tested by a binary that CONTAINS the fix it is about … So when that run's sink emits no
sink_capped, THAT IS NOT A FALSIFICATION OF F115. It is an UNCONTROLLED ZERO.
The marker idea is the one thing here that is unique to benchmarking a system you are simultaneously editing. On a fast-moving fork the binary under test is a moving target, and a prediction settled against a build that never contained the mechanism is not settled at all.
Withdrawals are recorded inline with their reasons, which is the part that makes the file trustworthy. One makespan falsifier was withdrawn because "measured 3-node detail makespan is 146.7/240.0/1112.9/1859.8s across identical configs, a 12.68x SPREAD". Another, a "1 to 3 per run" band, was withdrawn because "the archive shows 1/19, 4/22, 3/21, 3/17 … and I had that data when I registered it".
verdicts.py keeps three outcomes strictly apart, PASS, FAIL and INERT, on the grounds that "INERT IS NOT A PASS. A tool that prints two colours will eventually be read as if the third did not exist." Then it adds a fourth, and the reason is the best self-audit in the repo:
A FOURTH OUTCOME, added because three of the twelve entries in this file turned out to have NO FAIL BRANCH AT ALL … a quarter of the "12 checks, 23 cells, zero reds" column was green by construction rather than by evidence.
The payoff for all of this is tempo. Commit fbed900a2 records a falsifier firing 25 minutes after being registered and killing the campaign's own lead hypothesis: a 3-node cell scoring 0.7110 with one round-0 finding, six fix dispatches over three verify rounds, and complete_result{passed:false, remaining:1}, which is precisely the case that had been pre-registered as refuting. Dead: the causal reading, and with it the lead candidate. Also dead in the same message, and inconvenient: "six fixes gave 0.8986 on probe_post and 0.7110 here — fix count does not predict score." What survived was narrower and true.
I will close this section on the ledger honestly, because the flattering version is available and wrong. LEVERS.md is the audit sheet, one row per GOOSE_SWARM_* gate, under the rule that "A lever left undecided is P1 debt". I counted the verdict column myself: 114 rows, of which 3 carry a disposition, 37 carry a code-read or session-read verdict, and 74 carry only the value the engine itself emitted with an empty verdict cell. The loop is instrumented. It is not closed, and that is the more interesting sentence.
When one defect is reported as breadth, fix the report
Commit 84b86a1e6 opens: "Tier B is not twelve checks — it is one defect counted eight times, and the flaw is mine." Seven checks moved in lockstep across every cell, and their own detail strings say why: "0/247 payments after one sync", "no rows", "total=0", "too few rows", "total_minor=0", "oldest=None newest=None", "second sync inserted=0 total=0". Seven grammars, one sentence: there is no data. The four checks holding at 1.00 in every cell are exactly the four that need no synced data.
The consequence is a re-reading of a whole campaign. "Every 'Tier B is 52% of all score lost' line in this campaign must be read as 'the sync returning zero rows is 52% of all score lost'." One upstream defect costs a large multiple of what a genuinely independent tier B defect costs. An 8x weighting nobody chose.
Then the beat that makes this my favourite passage in the record. The obvious repair is to mark the blocked checks "unscored" rather than zero. Commit 256386ee5 is titled "my own fix for F752 was backwards — it would have PAID a broken app for failing early", and it shows the arithmetic: worked through, that change lifts a totally broken cell from 0.7226 to 0.8309. An app whose headline feature returns nothing would be rewarded for the checks it never got far enough to fail. "Absent input must score zero, never full marks" cuts both ways. Plan withdrawn.
What shipped instead is attribution with the score untouched. ROOT_BLOCKS names six tier-B dependents of sync_completeness, attribute_root_causes is pure and purely additive, and SCORER_VERSION is deliberately not bumped so the corpus needs no re-scoring and nothing already published moves. The output line survives in format_report under a comment that names its job, THE LINE THAT STOPS A TIER MEAN BEING READ AS BREADTH:
⚠ N further check(s) are downstream of `root`, which scored 0 — this is ONE defect, not N+1
Three details make it credible. The commit corrects its own arithmetic mid-flight: the root defect zeroes 7 tier-B checks, not 8, so the blast radius is 7/12 × 0.30 = 0.175 rather than 0.20. local_pagination is excluded from the blocked set on purpose because it scores 0.33 rather than 0.00 on the same cells, and "a partially-independent check does not belong in a blocked set". And the guard rootcheck.py holds the attribution in both directions against real stored verdicts.
Since it is a pure function, I ran it here rather than trusting the guard's report:
root=0, all 6 dependents=0 -> attributes exactly 6
root=1, all 6 dependents=1 -> attributes nothing
root=1, one dependent=0 -> attributes nothing (the defect keeps its own name)
root=0, 3 of 6 at 0.5 -> attributes only the 3 that are genuinely zero
The third line is the one that matters. An attribution that fires everywhere excuses real defects, which is the exact mirror of the error it was written to fix. The commit notes that it was "Watched refusing with the guard removed before being trusted green", and that generalises better than anything else here: watch a fix fail before you trust it passing.
The noise floor eats the effect
This is where a benchmark of a local swarm stops resembling a cloud one.
Commit dfaf078f2 did the arithmetic that should govern every future arm. Pooled score standard deviation on this fleet is 0.1767, giving a minimum detectable effect of 0.194 against an observed gap of 0.004. Detecting a 0.05 effect on score needs roughly 196 runs per arm; seeing 5 minutes of wall clock needs about 434. At roughly 105 minutes for a 3-node run, that is not a benchmark anyone is going to run. Both headline differences sat under one standard error: wall +8.11 ± 10.04, which is 0.81 SE, and score −0.0038 ± 0.0716, which is 0.05 SE.
The discipline that came out of it is the quotable part: "'Three nodes is not winning' is supported only as NOT MEASURABLE, never as LOSING — and I have repeatedly said the stronger thing." Along with the retraction of an earlier claim that a deficit had cleared one standard error: "A difference that crosses and re-crosses one SE on a single new row was never a finding, and I should have said so when it first crossed, not when it crossed back."
The escape route is to measure something with lower variance. Exactly one phase difference clears two standard errors, planning, at +13.36 minutes with SE 3.68 and t = +3.63. Research is a flat null. Everything after first dispatch is a null, with one term's sign flipping if a single row is dropped. The largest single effect in the engine is the planner's ladder excess: 41.04 minutes of planning when it fires against 14.52 when it does not, t = +9.6. I wrote about the practical end of that work in making the goose swarm predictable.
The companion rule from the same commit is a lever that reads as protection and cannot fire. GOOSE_SWARM_COMPLETE_STALL_ROUNDS needs stall >= 2, while its own round arithmetic can only ever reach 1. It was verified as "complete_stall_exit 0 times across 54 logs while the SAME reader on the SAME files finds complete_verify 94 and complete_fix_completed 71. A proven zero, not an observed one." That distinction is the whole of the positive-control discipline in one line.
Pair it with the physical confound, which cloud benchmarks never face. Commit a7d0d2f40: "F449's +0.1974 and 2.2x speedup is DEAD. It was measured with the 1-node arm pinned to the slowest machine … I was chasing a gap my own bug made." The honest re-run put 1 node at mean 0.8044 and 105 minutes against 3 nodes at 0.8536 and 172 minutes, which is 1.63 times slower, with the extra nodes buying about 1.6 times more code. Textbook Amdahl, and the wall ratio of 1.63 is almost exactly the tree ratio of 1.60. The lesson recorded there: when adding workers makes the job bigger, parallelism pays for itself only if the join scales too, so measure the join before claiming the fan.
I should be equally careful about what that does and does not license. Three nodes produce roughly 1.6 times more code and take roughly 1.6 times longer. They have not been shown to build a better app, and the pre-registered rule binds: if wall clock favours three nodes but score does not, the claim fails. Both, or neither.
The architectural punchline is my favourite method note in the project. Commit 1aa811ec4: at one node the fleet binds in 6 of 6 runs; at three nodes the plan binds in 9 of 10. Mean usable DAG width goes from 2.92 to 4.38, so tripling the hardware leaves roughly 1.6 of 6 slots with no work the plan can give them, "which makes every speed lever chased today second-order". That was found by reading a number the engine had been printing all along, across every archived run instead of one cell at a time. The discovery was an act of reading, not of instrumentation. The other hardware ceiling worth knowing about on this class of machine is the one I measured in 96 GB is 77.76 GiB.
Audit your own ruler
The arc completes with the instrument measuring itself, which is what turns a war story into a method.
SB6-DATA-CASE.md builds a corpus in public: 19,509 verdict.json files reduce to 309 unique verdict contents, then 214 with a live tree, then 93 unique builds after deduping 114 rescores of one identical tree, spanning 0.0167 to 0.9949. Then it measures the ruler against that corpus, and the numbers are not kind:
25 of 60 checks pass for at least 90% of serious entrants, so 42.8 points of the 100-point weight is guaranteed by "kinda works". At the 80% threshold it is 41 checks and 70.3 points.
32 of 60 checks produced two or fewer distinct values across all 93 builds.
44 of 60 sit at a mean of exactly 1.000 for the top cohort, so there is zero resolution above 0.90.
Mean total loss per top build is 2.76 points, and four of the five contributing checks are broken or noise.
The single most damning line is that Opus and Sonnet score identically on 57 of 60 checks, and the entire 0.63-point gap between them is client_timeouts, ui_currency and ui_polish. "The leaderboard order above 0.95 is decided by regex greps of the source, not by anything either app does." That is the bill for the static checks I flagged as weak earlier, and it arrived.
Three more failure classes worth looking for in your own sheet. v_status_distinct has a structural ceiling of 0.5 because the fixture sets every payment to settled, so the check "currently punishes the fixture". The performance tier, in the data case's own words, is "binary aliveness checks wearing a stopwatch": p_list_latency measures a p95 of 0.24 to 0.74 ms against a 150 ms budget, which is 203 to 625 times more slack than the budget asks for. And request_efficiency has the live vacuous pass I reproduced above.
What that audit bought is the next scorer. score_sb6.py is 2,684 lines as of the branch I read today, with 69 registered checks across ten codes, of which nine carry weight zero as DIAGNOSTIC under an explicit anti-stacking rule, leaving 60 that can move the number. The formula is score = 0.88 × inner + 0.12 × gate_fraction × e_mean, and compound checks take the min of their components rather than the product, because "the weakest component bounds the whole, so one defect stops stacking partial credit". The T and HARD tiers together are 0.34 of the inner mean, which is close to 30% of the published number once the 0.88 is applied.
Two mechanisms in there are worth copying wholesale. The first is tamper evidence:
assert TIER_WEIGHT_SB6["T"] + TIER_WEIGHT_SB6["HARD"] >= 0.34, "hard-axis weight floor"
assert E_WEIGHT >= 0.10, "excellence slice floor"
Any future edit that re-compresses the instrument has to delete an assert to do it, and the diff says so out loud. The second is the sha pin. sb6-thresholds.json is frozen and marked calibrated: true, and a file claiming that whose hash does not match the constant baked into the scorer makes the scorer refuse to score, as a structural refusal rather than a silent default. I hashed the file myself rather than trusting the note:
$ shasum -a 256 sb6-thresholds.json
723e71b5a029885103b2a4c61674bc8c8c6ca415ac397e20f3f7ecd1aa66490a sb6-thresholds.json
That is byte-for-byte the value pinned in score_sb6.CALIB_SHA256. Worth reading the file's own caveat in the same breath, because it is the honest part: within that frozen fit the three global shaping knobs all sit at identity 1.0, and the note says plainly "n=1 per cloud model".
And the doctrine that pays for itself every run:
avail = [r for r in sub if not r.get("unavailable")]
Unavailable rows never become application zeros. The tier mean is taken over what was actually measured, and the verdict names every excluded row out loud. That is the absence-versus-zero rule from the very first section, arriving in the arithmetic.
The board has since moved on again; the current era there is sb-7, and each era is shown on its own because cross-era comparison is not meaningful. Every check in every live era is documented individually on the methodology page, each era with the exact formula that produced its numbers.
What to take home
Prove the lever moves before you A/B it. Make the system emit its own resolved settings, computed by calling the same expressions it branches on. Anything that re-derives them will eventually disagree, and a harness printing what it set is not evidence of what the process read.
Spell absence and zero differently. A field missing from an event must not read as resolved-to-null, and a probe that failed must not read as an application that failed.
Grade in [0,1], never binary. Three of 47 and 46 of 47 are not the same result, and a sheet that cannot say so cannot resolve the distance you care about.
Make every deduction name its harm. If you cannot name what goes wrong in the real world, cut the check.
Prove the grader in both directions, and detect defects by delta against the known-good's own vector rather than by membership in a list.
Watch a fix fail before you trust it green. A guard that has only ever printed green is indistinguishable from one that cannot print anything else.
Positive-control every zero. Count the files your fixer opened. A zero you have proved the counter could see is evidence; a zero you merely observed is a shrug.
Pre-register the falsifier and the marker. A check with no fail branch is an observation, not a verdict, and dressing it as a pass inflates your apparent health.
When the number is right but misread, fix the report rather than the weight. Marking blocked checks unscored pays a broken app for failing early.
Score serially, into a wiped tree, at the advertised port. Machine load, concurrency and port are all terms in a benchmark number, and not one of them is the entrant.
If you build one thing from this article, build the audit rather than the scorer. Take your own corpus of graded results, compute per check the pass rate and the number of distinct values it has ever produced, and find out how many of your points are being handed out for "kinda works". Here the answer was 42.8 of 100, and I did not know that until I looked.
The whole of it is readable at leanzero-srl/goose-local-edition under evals/swarm-bench/: the scorer, the controls, the mock vendor and the prediction register. The commands in this article are the ones I ran. If you find a place where the ruler is still flattering itself, I would genuinely rather hear it than not.
