A New Cheap Model Dropped This Week. Here's the Harness I Run Before I Switch Anything

Every few weeks a new model release hits my feed with the same pitch: cheaper, faster, just as good. Sometimes that's true for someone's workload. It is rarely true for mine until I check, because "as good" is always measured on someone else's tasks.

I've written before about benchmarking coding models with a fixed task suite instead of vibes. This post is the operational follow-up: what I actually do in the 48 hours after a hyped release, before I let it anywhere near real work. The short version: I run a small regression harness on a throwaway server, compare the new model against my current default on my tasks, and only then decide what gets routed where.

Why not just try it on real work

Two reasons:

1. Failures are silent. A model that's 15% worse at following output-format constraints doesn't announce itself. It quietly produces diffs that look fine and break CI later.
2. First impressions are anchored by price. Knowing something is cheap makes me grade it leniently. A blind-ish, scripted comparison removes that.

So the rule in my setup is: no model earns production traffic without passing a fixed suite first. Hype doesn't get a waiver.

The setup: a free box, a free model tier, zero risk

You don't need paid infrastructure for this. Eval runs are bursty and low-stakes, which makes them a great fit for free tiers. I run my harness on MonkeyCode, which offers free model access and a free server option — enough to host a small eval runner and poke candidate models without touching my paid quotas.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The architecture is boring on purpose:

• One server process that exposes a uniform complete(prompt) -> text interface per model.
• A fixed task suite stored as JSON files, versioned in git.
• A runner that executes every task against two endpoints (incumbent vs. challenger) and writes results to disk.
• A scorer — partly programmatic, partly me reading diffs.

Because the server is disposable, I can also point the harness at deliberately hostile inputs (prompt injection in file contents, misleading comments) without worrying about what the model does with elevated access. Same sandbox logic I wrote about previously: break things where breaking is free.

The artifact: a minimal two-model diff harness

Here's the core runner, trimmed but runnable. It assumes both models sit behind an OpenAI-compatible chat endpoint, which most providers and self-hosted gateways offer:

import json, time, pathlib, urllib.request

MODELS = {
    "incumbent": {"url": "http://localhost:8000/v1/chat/completions", "model": "current-default"},
    "challenger": {"url": "http://localhost:8001/v1/chat/completions", "model": "new-release"},
}

def complete(cfg, prompt):
    body = json.dumps({
        "model": cfg["model"],
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        cfg["url"], data=body,
        headers={"Content-Type": "application/json"})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        out = json.loads(r.read())
    return out["choices"][0]["message"]["content"], time.time() - t0

def run_suite(suite_dir, out_dir):
    for task in sorted(pathlib.Path(suite_dir).glob("*.json")):
        spec = json.loads(task.read_text())
        row = {"task": task.stem, "rubric": spec["rubric"]}
        for name, cfg in MODELS.items():
            try:
                text, secs = complete(cfg, spec["prompt"])
                row[name] = {"output": text, "secs": round(secs, 2)}
            except Exception as e:
                row[name] = {"error": str(e)}
        pathlib.Path(out_dir, task.name).write_text(json.dumps(row, indent=2))
        print(f"{task.stem}: done")

The suite itself matters more than the runner. Mine has 20 tasks in five buckets, all drawn from real work I did in the last month (sanitized):

Each task has a rubric — a checklist I score by hand. Fully automated scoring tempts you into grading with another model, which re-imports the vibes problem through the back door. I automate collection and keep judgment manual. It takes me about 40 minutes to score a 20-task diff, once.

How I read the results

Three passes:

1. Hard failures first. Any task where the challenger produced invalid output (unparseable JSON, patch that doesn't apply) is a red flag regardless of how "smart" the text reads. Two or more hard failures = not this week.
2. Rubric diff. I score incumbent and challenger side by side without looking at which is which until the end. (I label the files by hash and reveal after scoring. Crude, effective.)
3. Cost-adjusted decision. Only if quality is within noise do I look at price. A model that's 5% worse and 60% cheaper might win for a specific route — like drafting commit messages — while losing the default slot.

That last point is the real takeaway: the answer is rarely "switch everything." It's usually a routing table. New releases earn the low-stakes routes first and get promoted if the suite keeps passing on fresher tasks.

Limitations and who shouldn't bother

• 20 tasks is a smoke test, not a benchmark. It catches regressions in my distribution of work. It says nothing about yours. Build your own suite from your own history; copying mine defeats the purpose.
• Temperature 0 doesn't make models deterministic. Same harness, next Tuesday, slightly different outputs. I re-run the suite before any promotion decision, not just once.
• Free tiers have ceilings. Rate limits and queueing are fine for a 20-task nightly run and wrong for latency measurement. Don't benchmark speed on a free tier and expect the numbers to transfer to paid capacity.
• If your workload is one prompt repeated at scale (classification, extraction), a fixed golden set with automated exact-match scoring is a better tool than this whole setup.

The actual workflow, condensed

1. New model drops; feed gets excited.
2. I spin up a throwaway server, wire the challenger endpoint into the harness.
3. Nightly run against my versioned suite; I score the diff with coffee the next morning.
4. Pass → gets a low-stakes route. Fail → I write down why and retest after the next release.

If you want to try this without spending anything while you iterate, MonkeyCode's free model access and free server are a reasonable place to host the harness — the whole point is that evaluation infrastructure should cost less than your curiosity.

The models will keep coming weekly. The harness is the part that compounds.