Make Remote Eval Workspace a Hash-Fenced Snapshot Before Late Scores Stick

I was reviewing a planner design last Tuesday when a sequence trace showed a green score landing on a rewritten step. A remote worker had started pytest against generation seven, stalled hard, then posted success after generation eight already existed. Does that failure look like a weak model to you, or does it look like a missing snapshot protocol around the checkout? I am writing this as an architecture review of constraints, data flow, and failure domains, not as an operations runbook.

Most pipelines farm coding evals to a remote box and then store the first JSON blob that resembles a test summary. That pattern assumes the worker still holds the workspace the planner intended to score, which is a lie under preemption. I want a tighter invariant than job completion: a verdict may attach only when its tree hash still equals the step's current workspace generation.

The violating event order

I keep returning to one counterexample because it is small enough to simulate and still wrecks a scoreboard.

1. The planner writes workspace generation seven and records tree hash H7 before dispatch.
2. Worker W1 leases that job and starts the test command against the H7 tree.
3. The planner times out, compensates, and writes generation eight with a new hash H8.
4. W1 finally wakes and uploads a pass record keyed only by job id, and the store overwrites the live step.

Which field in that JSON payload actually names the workspace tree you believe you scored just now? If you answer "the job identifier," you already lost, because identifiers survive generation bumps without describing contents. The invariant the common store fails to preserve is verdict.tree_hash == step.workspace_hash at attach time, otherwise the write must be rejected.

Declared assumptions

I assume a planner that owns one mutable checkout per eval step and farms tests onto an untrusted remote worker process. I assume that worker may be preempted, retried, or reused, and that wall clocks across hosts are not a shared truth. I assume the planner can hash the workspace before dispatch and that the score store can refuse a compare-and-swap. I do not assume exactly-once delivery, sticky leases, or a dedicated machine per generation.

Why am I this loud about assumptions before showing a diagram? Because a remote host is a separate failure domain even when you launched it from the same repo. If you skip the hash check, you are trusting scheduler luck to preserve correctness.

Data flow and failure domains

sequenceDiagram
    participant P as Planner
    participant S as Score store
    participant W as Remote worker
    P->>S: record gen=7 hash=H7
    P->>W: dispatch job, gen, hash
    Note over P: timeout, compensate
    P->>S: record gen=8 hash=H8
    W->>S: late pass for gen=7
    S-->>W: reject_stale
    P->>W: dispatch gen=8
    W->>S: pass for H8
    S-->>P: attached

I split three failure domains on purpose so the review does not collapse into a single happy path. The planner can crash after bumping generation and leave a live worker holding a dead tree. The score store can accept a write the planner would have rejected if it still sat on the request path. The remote worker can sleep for minutes and then finish work against a checkout that no longer exists.

Ask yourself which domain your current design treats as source of truth for pass and fail. If the worker is canonical, a preempted process can rewind the step after you already compensated. If the planner is canonical but the key is only a job id, a retry still collides on the same slot.

Constraints I refuse to relax

Latency of a coding eval is dominated by dependency install and test runtime, not by comparing a thirty-byte hash. Throughput is dominated by how many workspaces you can isolate, not by how quickly you acknowledge a JSON body. Correctness means a green score never attaches to a rewritten tree, even when the late worker was honest. Cost may include retrying compute, but it must not include double-counting two generations as one truth.

I would rather drop a late pass than publish a score for a snapshot the current step does not contain. Would you rather explain a rejected verdict to a dashboard, or explain why a shipped step never saw the tests you filed?

Build the fence in five numbered steps

Follow this path if you want a protocol you can property-test instead of a slide about idempotent jobs.

Step 1. Snapshot the tree on the planner

Hash the workspace in the planner process before dispatch, not on the worker after it mutates files. I include tracked contents plus a generation integer so retries cannot impersonate an older snapshot by accident. The worker receives the expected hash as an opaque token that it must echo on completion.

Step 2. Bind the lease to generation and hash together

A job record should look like step identifier, generation, tree hash, and lease deadline as one tuple. Completing the job requires echoing every field, not just presenting a still-valid job id. This is a protocol check, and it is not a retry backoff policy in disguise.

Step 3. Attach through compare-and-swap only

The store attaches a verdict only when current generation equals the verdict generation and both hashes match exactly. A late worker receives rejected_stale instead of a silent overwrite of the live step. Should you page someone for that rejection? I would not, because the fence did the job.

Step 4. Compensate by bumping generation, not by recycling the slot

When the planner times out, it increments generation and writes a new hash rather than deleting the old job identity. Old workers may still finish, and that outcome is acceptable when the store refuses their payloads. Reusing the same slot is how a green rewind sneaks into the scoreboard.

Step 5. Inject the late-wakeup path inside the harness

Your evaluation harness is incomplete until it can pause worker one, bump generation, and then deliver the stale payload. If that path exists only in production traffic, you do not have an evaluation system yet. You have a scheduler that has been lucky.

Minimal simulator

The fixture below is a proposed state machine for the counterexample, not a production score service with measured throughput. I want the late pass to bounce, and I want the on-time fail to attach to generation eight.

from dataclasses import dataclass
from typing import Optional, List, Tuple

@dataclass
class Step:
    generation: int
    tree_hash: str
    verdict: Optional[str] = None

@dataclass
class Verdict:
    generation: int
    tree_hash: str
    result: str

def attach(step: Step, v: Verdict) -> str:
    if v.generation != step.generation:
        return "reject_generation"
    if v.tree_hash != step.tree_hash:
        return "reject_hash"
    if step.verdict is not None:
        return "reject_duplicate"
    step.verdict = v.result
    return "attached"

def simulate_late_wakeup() -> Tuple[List[str], Optional[str]]:
    step = Step(generation=7, tree_hash="H7")
    late = Verdict(generation=7, tree_hash="H7", result="pass")
    step.generation = 8
    step.tree_hash = "H8"
    events = [attach(step, late)]
    on_time = Verdict(generation=8, tree_hash="H8", result="fail")
    events.append(attach(step, on_time))
    return events, step.verdict

def simulate_hash_drift() -> str:
    step = Step(generation=8, tree_hash="H8")
    drifted = Verdict(generation=8, tree_hash="H7", result="pass")
    return attach(step, drifted)

if __name__ == "__main__":
    events, verdict = simulate_late_wakeup()
    assert events == ["reject_generation", "attached"]
    assert verdict == "fail"
    assert simulate_hash_drift() == "reject_hash"
    print("invariant held", events, verdict)

Run it with python fence_sim.py from any working directory that can import the stdlib. If attach returns attached for the late payload, the invariant is already dead and you should not scale the worker pool. I also want a second command that prints the drifted-hash path so reviewers cannot claim only timeouts were tested.

python - <<'PY'
from fence_sim import simulate_hash_drift
print(simulate_hash_drift())
PY

Failure injection

I inject four failure classes because each one belongs to a different domain in the diagram.

1. Preempt and resume: the worker pauses after tests and resumes after a generation bump.
2. Duplicate delivery: the same matching verdict arrives twice against the live hash.
3. Hash drift: the worker mutates the tree and still reports the original generation integer.
4. Clock lie: the planner expires the lease while the worker still believes it owns the job.

The property I will accept is narrow and testable. After any injected class, step.verdict either matches the current hash or stays empty, and it never equals a previous hash. Can your queue consumer state that property as code, or does it live only as a comment beside the ACK?

Tradeoff table

I pick hash plus generation for a single-writer checkout, and I would not stretch it over a multi-writer monorepo without a real commit fence. The explicit denominator is scored steps whose workspace hash still matches at attach time, not jobs that returned HTTP 200. Acceptance rule: in one hundred injected late-wakeup runs, zero stale passes attach, and one attach means the design does not converge.

What I would change next

I would stop treating the remote worker as a scorer and start treating it as an untrusted producer of candidate artifacts. The planner should pull a tarball, rehash the tree, and only then commit a tiny local verifier, even when heavy tests already ran remotely. That split keeps compute in the cheap domain and keeps commit inside the planner's failure domain.

I would also version the verdict schema so a missing tree hash is a protocol violation instead of a defaulted pass. An empty 200 is not success, and pytest JSON should not get a weaker contract than tool calls already required. Next after that, I would record rejected stale verdicts as first-class events so eval dashboards stop calling fences "worker errors."

Where a spare remote worker fits

You cannot honestly review this design if both processes share one laptop scheduler and one memory space. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is enough to stand up a disposable worker that can stall and post a late payload.

I would park the score store on the planner side and use that free server only as worker W1 in the sequence. I would use free model access to draft additional property cases against attach(), not to decide whether a stale hash may stick. The point is cheap preemption rehearsal, and it is not a durability guarantee.

If that domain split helps you, run the fixture against a worker you do not control and watch which event order your store currently swallows.

Limitations and who should skip this

Skip this fence if your eval already runs in a hermetic single-process harness with no remote mutation at all. You would be adding protocol surface for a failure domain you do not actually have. Skip it if you cannot hash the workspace, because the echoed token becomes theater and the compare-and-swap protects nothing.

Do not treat free remote capacity as a durability contract, a quota promise, or a substitute for a commit log. This is a snapshot protocol for single-writer coding evaluations, and it is not a saga, a quorum, or an incident playbook. People chasing model quality without a score store should fix the harness first.

Validate the conclusion

Clone the simulator and add a fifth event where the worker reports generation eight with tree hash H7. Should the system reject that row, replay the job, or compensate in place? I want a reject, then a planner-owned replay on a fresh snapshot, and I never want a silent compensate that reuses the stale hash.

Which event order in your pipeline still writes a pass without looking at the tree, and will you reject, replay, or compensate when it shows up?