A verifier should add evidence, not rewrite history

I spent the last few weeks closing a loop around an AI coding agent.

The first version had an obvious flaw: the worker could see the command and expected output that would grade its work. A stuck agent could optimize for the assertion string instead of the requirement. PR #1916 removed that answer key from the worker-facing success contract and connected one invocation through run, evaluation, and bounded evolution. RFC #1917 records the design.

That change removed the answer key, but it exposed a separate design question: how much authority should the verifier have over facts that already happened?

A worker's completion status and a later machine check are separate facts. The worker either completed the acceptance criterion or it did not. The machine check may then produce supporting evidence, find a real failure, or fail to run. Collapse those facts into one boolean and the loop can report an outcome that never occurred.

Ouroboros v0.51.12 uses one rule at that boundary:

Machine verification is monotonic: it adds evidence to a successful worker result, but it cannot recover a failed worker result or turn completed work into a retry when the verifier is unavailable.

The implementation is PR #2187. The release is v0.51.12.

The four outcomes

The rule has four outcomes:

A passing command cannot rescue failed work

Earlier versions treated the presence of a verify_command as a reason to relax some transcript evidence. That created a perverse result: adding exit 0 could make an acceptance criterion weaker than one with no command.

The current evidence classifier discards verifier-specific exemptions. Its docstring states the invariant directly:

A verify command is additive evidence. Declaring one never removes the
worker transcript obligations; this keeps a vacuous command from making
an acceptance criterion weaker than a contract-less one.

When the active execution profile requires files_touched, commands_run, or tests_passed, each still needs its own support. The verify command runs afterward and adds one more observation.

The execution path enforces that order. _run_ac_verify_gate is called only when the worker result is already successful:

if success and verify_gate_active and has_success_contract:
    verify_gate_outcome = await _run_ac_verify_gate(...)

A green command cannot convert failed work into success because the command does not run in that branch.

Missing verifier infrastructure cannot manufacture new work

Suppose the worker completed the task, but the machine has no Bash, the verifier process cannot start, or it times out before producing a judgment.

A worker retry fixes none of those conditions. It only repeats completed work to compensate for infrastructure the worker does not own.

v0.51.12 records that case as UNAVAILABLE. The typed contract is strict:

UNAVAILABLE requires TRANSCRIPT_MISSING_INFRASTRUCTURE and ACCEPT

ACCEPT permits execution to continue without retrying the worker; it does not claim that verification passed. The final report carries the result as unverified success.

Recording an unavailable verifier as worker failure would send the loop back into code generation, where it could overwrite correct work while leaving the missing verifier untouched.

A real failed check still rejects the result

A resolved verifier that runs and returns a nonzero status rejects the successful worker result:

if returncode != 0:
    return VerifyGateOutcome(
        passed=False,
        reason=f"verify_command exited with status {returncode}",
    )

An output_assertion missing from the combined output also rejects the result. Real negative evidence can move a successful result to rejected. It cannot move a failed worker result upward or retry the worker because the verifier could not judge.

The state transitions are:

worker failure ------------------------------> failure
worker success + verifier failure ----------> rejection
worker success + verifier pass -------------> verified success
worker success + verifier unavailable ------> unverified success

The shell is part of the judgment boundary

A verification command is only meaningful if the runtime executes the command that the acceptance contract declared.

The new path resolves an absolute Bash implementation and executes the original text through bash -c. It does not translate the command for cmd.exe, substitute sh, invoke the WSL launcher against another filesystem, or fall back to a shell emulator.

The command text stays unchanged. If a compatible Bash does not exist, the outcome is UNAVAILABLE rather than an approximation presented as evidence.

The process boundary also strips environment controls that can alter a verdict without changing the command or workspace. The list includes:

• BASH_ENV, exported Bash functions, shell option state, and compatibility controls
• PYTHONPATH, PYTHONSTARTUP, and PYTHONHOME
• PYTEST_ADDOPTS and PYTEST_PLUGINS
• NODE_OPTIONS
• dynamic-loader preload hooks

PYTEST_ADDOPTS="--collect-only" is a small example. A test command can exit successfully without running the tests the contract author thought it ran. An exported function named pytest can replace the executable entirely. Both are outside the command text, so the verifier removes them before judgment.

Timeout and cancellation handling contain the verifier process tree. On POSIX, the process starts in a new session. On Windows, it is assigned to a kill-on-close Job Object before it is resumed. A timed-out check should not leave a background child mutating the workspace after the verdict has been recorded.

What this still does not prove

A real Bash process and a sanitized environment do not create a sandbox.

The workspace can still contain conftest.py, pytest.ini, a project-owned virtual environment, or a test that passed before the requested change existed. A command such as test -f out.txt proves that a path exists, not that the artifact satisfies the acceptance criterion.

The RFC keeps negative-control and differential-probe ideas as future evaluation research. They stay outside the ooo run critical path because arbitrary probes need real isolation. Running destructive baseline commands against the live workspace would give a verifier more authority while making its evidence less trustworthy.

v0.51.12 therefore limits the critical path to claims the current runtime can defend:

1. preserve the worker's execution facts
2. require transcript evidence independently
3. run the declared command through a sealed real Bash
4. reject on real negative evidence
5. report UNAVAILABLE when no judgment occurred

Why this closes the earlier loop work

1916 and #1917 established three properties: the worker does not receive its criterion's answer key in the success contract, the orchestrator owns the judgment, and rejected results can feed a bounded run-to-evaluate-to-evolve chain.

Together, those properties make a convergence loop possible. #2187 limits what the judge may rewrite once that loop is running.

Exposing the answer key lets the agent optimize for it. Letting the verifier rewrite outcomes distorts the execution history. In either case, the score can improve while the evidence gets worse.

For agent loops that execute real work, I would use this ownership rule:

• generation owns attempts
• verification owns additional evidence
• infrastructure failure owns neither

If you maintain an agent loop, take one completed run and trace all four cases through your state machine. Verify that downtime does not retry the worker, a passing command cannot erase a failed execution, and adding a command does not remove existing evidence obligations.

Source: Q00/ouroboros, PR #2187, v0.51.12.