A No-Cost Harness for Comparing Free Coding-Agent Models and Runtimes

You can waste hours comparing coding agents by reading model cards or watching polished demos. The cheaper signal is a tiny audit that records whether the agent changes only the intended files and leaves the test suite green after one under-specified, realistic task. This guide gives you the harness, the fixture pattern, and the reading rubric.

MonkeyCode is a useful platform for this because its free model access lets you repeat the same probe across different model labels without metering every call, and its free server option removes your laptop's memory and GPU as the hidden variable. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What you are really comparing

You are not measuring whether the model can solve a clean LeetCode prompt. You are measuring whether it can make a contained change in a disposable Git directory without producing collateral damage. The audit records five signals: the agent command's exit code, elapsed time, modified files, the tail of the agent's stdout and stderr, and the test suite's exit code after the change. Those five signals are boring, and that is the point; they are cheap to collect and hard to argue with.

You should repeat the same fixture against different runtimes. A local run tells you how the agent behaves when it shares your machine's dependencies and network rules. A server-backed run tells you how much of the result depends on a controlled environment rather than your laptop. The comparison is only meaningful if the fixture and the test command stay identical across runs.

The harness

The Python script below clones nothing and assumes the agent writes into the current directory. It seeds a fixture, injects the task through the PROBE_TASK environment variable, runs the agent command, then reports the results as JSON.

#!/usr/bin/env python3
"""Run one coding-agent probe in a disposable Git directory."""
import argparse
import json
import os
from pathlib import Path
import subprocess
import tempfile
import time


def run(cmd, cwd, env):
    return subprocess.run(
        cmd,
        cwd=str(cwd),
        shell=True,
        capture_output=True,
        text=True,
        env=env,
    )


def seed(tmp, fixture):
    for source in fixture.rglob('*'):
        if source.is_file():
            dest = tmp / source.relative_to(fixture)
            dest.parent.mkdir(parents=True, exist_ok=True)
            dest.write_text(source.read_text())
    run('git init -q && git add -A && git commit -q -m seed', tmp, os.environ.copy())


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--fixture', required=True, type=Path)
    parser.add_argument('--agent', required=True, help='command; task is in $PROBE_TASK')
    parser.add_argument('--test', default='pytest -q', help='test command')
    args = parser.parse_args()

    with tempfile.TemporaryDirectory() as tmp_dir:
        root = Path(tmp_dir)
        seed(root, args.fixture)

        env = os.environ.copy()
        env['PROBE_TASK'] = (root / 'task.txt').read_text()

        start = time.time()
        agent = run(args.agent, root, env)
        elapsed = round(time.time() - start, 2)

        status_after = run('git status --porcelain', root, env).stdout
        diff_stat = run('git diff --stat', root, env).stdout
        tests = run(args.test, root, env)

        report = {
            'agent_returncode': agent.returncode,
            'elapsed_seconds': elapsed,
            'agent_stdout_tail': agent.stdout[-2000:],
            'agent_stderr_tail': agent.stderr[-2000:],
            'files_changed': status_after,
            'diff_stat': diff_stat,
            'test_returncode': tests.returncode,
            'test_stdout_tail': tests.stdout[-2000:],
        }
        print(json.dumps(report, indent=2))


if __name__ == '__main__':
    main()

Create a minimal fixture directory with three files. Keep the task deliberately under-specified so you can observe whether the agent asks for clarification or guesses and touches too much.

fixtures/parser/
├── task.txt
├── src/
│   └── parser.py
└── tests/
    └── test_parser.py

Add input validation to parse_record. Reject empty strings.

# src/parser.py
def parse_record(text):
    return text.split(',')

# tests/test_parser.py
import pytest
from src.parser import parse_record


def test_rejects_empty_string():
    with pytest.raises(ValueError):
        parse_record('')

Run the harness with the command shown in your account. The exact MonkeyCode flag name may differ, so treat this as a template rather than a contract.

python probe.py \
  --fixture fixtures/parser \
  --agent 'monkeycode run --task "$PROBE_TASK"' \
  --test 'pytest -q tests'

Model labels change often, so treat names such as DeepSeek-V4-Pro-0813 or Grok 4.6 as examples to verify in your account rather than facts about current availability. If a label appears in your account list, pass it through the agent command and keep the rest of the fixture identical. That is how you turn a marketing claim into a repeatable observation on your own code.

Comparing local and server-backed runs

If your account exposes a free server option, run the same fixture through that path. The harness does not care where the agent executes; it only cares about the files and test result in the disposable directory.

python probe.py \
  --fixture fixtures/parser \
  --agent 'monkeycode run --server free --task "$PROBE_TASK"' \
  --test 'pytest -q tests'

You should not compare wall-clock time directly between local and server-backed runs unless you also record queue latency, container startup, and network transfer. If those details are not visible to you, compare the output signals that have clear pass or fail meaning: which files changed, whether the test passes, and whether the agent returned an error.

Reading the report

• agent_returncode != 0 means the command failed before or during the change. Treat that as an availability or invocation problem first, not a model-quality result.
• test_returncode != 0 means the patch did not satisfy the fixture's expected behavior.
• Files outside src/ and tests/ are scope drift, especially new lock files, generated config, or unrelated modules.
• A very short elapsed time with a red test suite may mean the agent guessed instead of attempting the change.
• Output tails often reveal whether the agent asked for clarification, assumed a schema, or silently skipped work.

A single green patch is not proof of reliability. It is a cheap filter: if an agent cannot pass a two-file fixture with clear acceptance criteria, you do not need a larger benchmark to postpone using it for real work.

Limitations and who should not use this

This harness does not measure code quality, security, or production readiness. It only tells you whether the agent can make a bounded change in a disposable directory without leaving red tests. Availability claims also change faster than documentation, so verify the model labels and runtime flags before scripting a pipeline around them.

You should not use this approach with proprietary credentials, regulated data, or code where copying into a server-backed runtime would violate policy. You should also skip it if you need benchmark-style rankings; a five-minute boundary probe is not a ranking, just a poor producer's smoke test. Start with the cheapest available model label and a two-file fixture, then change one variable at a time. That red and green flip on your own fixture is usually more informative than another model card.