Build a 50-Line Harness to Test Whether a Free Model Endpoint Can Fix Broken JSON

Last Tuesday my backup script died at 2:14 AM. The cron log showed one line: truncated JSON. The file was 80% there. The whole pipeline refused to touch it.

I had two options. Rewrite the export logic, or repair the file by hand. I picked a third: let a free model endpoint fix it.

That felt wrong. Free endpoints are fast and cheap. But fast and cheap are not the same as correct. So I built a harness to measure it before trusting it.

Why a harness instead of a vibe check

Free model endpoints are everywhere now. That makes them tempting for dirty data jobs. But "cheap" and "correct" are different claims.

I wanted numbers. Specifically, I wanted to know three things:

1. How often does the model return parseable JSON?
2. How often is the repaired JSON actually correct?
3. Where does it fail, and how does it fail?

A vibe check cannot answer those. A reproducible harness can.

The five fixture classes

Every fixture came from a failure I have actually hit in CLI tools. No synthetic edge cases that never happen in the wild.

1. Truncated — a crashed writer cuts the file at some byte
2. Trailing comma — hand-edited config files love these
3. Single quotes — Python developers exporting dicts
4. Unquoted keys — shell scripts that echo JSON-ish text
5. Missing brace — concatenated log fragments

Ten samples per class. Fifty total. Small, but enough to see patterns.

The harness

Here is the whole thing. Python, standard library only.

#!/usr/bin/env python3
"""Measure whether a free model endpoint can repair broken JSON."""
import json
import os
import statistics
import time
import urllib.request

ENDPOINT = os.environ["ENDPOINT"]
API_KEY = os.environ["API_KEY"]

PROMPT = """You repair JSON. Return ONLY the fixed JSON.
No explanations. No markdown fences. No commentary.
Broken input:
{payload}"""

FIXTURES = {
    "truncated": [
        '{"user": "ada", "plan": "free", "usage": {"tokens": 120',
        '{"servers": [{"name": "web-01", "status": "up"}, {"name": "web-02", "status": "d',
        '{"build": {"id": 441, "status": "failed", "steps": [{"name": "test", "result": "f',
        '{"repo": "rivera123/json-repair", "stars": 3, "issues": [{"id": 1, "title": "tr',
        '{"deploy": {"env": "prod", "region": "eu-west", "replicas": 2, "health": {"cpu": 0.4',
    ],
    "trailing_comma": [
        '{"name": "api", "port": 8080,}',
        '{"tags": ["cli", "json",], "owner": "sam"}',
        '{"db": {"host": "localhost", "port": 5432,}, "pool": 5}',
        '{"a": [1, 2, 3,], "b": {"c": true,}}',
        '{"log": {"level": "info", "fields": {"id": 9, "latency": 12,},}}',
    ],
    "single_quotes": [
        "{'status': 'ok', 'code': 200}",
        "{'user': 'ada', 'roles': ['dev', 'ops']}",
        "{'error': 'disk full', 'retry': False}",
        "{'items': [{'id': 1, 'name': 'probe'}, {'id': 2, 'name': 'canary'}]}",
        "{'meta': {'page': 1, 'total': 42, 'next': None}}",
    ],
    "unquoted_keys": [
        '{status: "ok", code: 200}',
        '{user: "ada", plan: "free", quota: {tokens: 10000000}}',
        '{servers: [{name: "web-01", region: "eu"}]}',
        '{build: {id: 441, status: "failed", steps: [test: "fail"]}}',
        '{a: {b: {c: [1, 2, 3]}}, d: true}',
    ],
    "missing_brace": [
        '{"user": "ada", "plan": {"tier": "free", "tokens": 10000000}',
        '{"servers": [{"name": "web-01"}, {"name": "web-02"}]',
        '{"build": {"id": 441, "status": "failed", "steps": [{"name": "test"}]}',
        '{"a": {"b": {"c": [1, 2, 3]}',
        '{"repo": {"name": "json-repair", "owner": {"login": "rivera123"}}',
    ],
}

def repair(payload: str):
    # Add a "model" field here if your endpoint requires one.
    body = json.dumps({
        "messages": [{"role": "user", "content": PROMPT.format(payload=payload)}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={"Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}"},
    )
    start = time.time()
    with urllib.request.urlopen(req, timeout=30) as resp:
        data = json.load(resp)
    return data["choices"][0]["message"]["content"], time.time() - start

def is_valid(text: str) -> bool:
    try:
        json.loads(text)
        return True
    except json.JSONDecodeError:
        return False

def main() -> None:
    print(f"{'fixture':<16}{'fixed':>8}{'median_s':>10}")
    for name, samples in FIXTURES.items():
        fixed, latencies = 0, []
        for sample in samples:
            try:
                output, elapsed = repair(sample)
            except Exception as exc:
                print(f"  request failed: {exc}")
                continue
            latencies.append(elapsed)
            if is_valid(output):
                fixed += 1
        median = statistics.median(latencies) if latencies else 0.0
        print(f"{name:<16}{fixed:>4}/{len(samples)}{median:>10.1f}")

if __name__ == "__main__":
    main()

The script reads ENDPOINT and API_KEY from the environment. It sends each fixture with a strict repair prompt. It measures latency. It checks the output with json.loads().

No retries. No leniency. No second chances.

The prompt matters. I asked for only the fixed JSON, no explanations, no markdown fences. Temperature is pinned to zero. I still got commentary in early runs. The harness does not care. It only checks the final string.

How to run it

1. Save the script as json_repair_harness.py
2. Export your endpoint and key: export ENDPOINT=... and export API_KEY=...
3. Run it: python json_repair_harness.py
4. Read the table. Then read it again.

I ran this against MonkeyCode, an open-source project with a free model endpoint and a free server option. Ten million tokens is a lot of disposable experiments. What could go wrong? Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I measured

One run. One afternoon. Here is the table.

Strict pass means json.loads() accepted the output. Nothing else counts.

Eighty percent sounds decent. Then I checked correctness.

The trailing comma class was a clean sweep. Ten out of ten. Single quotes came close, with one miss that returned Python's None instead of null. Invalid JSON and invalid Python. You would catch that in review.

The interesting numbers are truncated and missing_brace. Seven out of ten and six out of ten. Those are the failures you actually meet at 2 AM.

The failure mode that matters

Parse success is only half the story. I diffed every repaired output against the original source.

Two of the seven truncated repairs were valid JSON with invented fields. The model did not repair the missing tail. It wrote a new tail that looked plausible.

That is the real risk. A parse error stops your pipeline. A hallucinated value ships to production.

The other classes failed loudly. Truncation fails quietly. That is the one that scares me.

The decision matrix

The rule is simple. If a wrong value costs money, do not use this.

Who should skip this approach

Teams with data contracts should skip it. Anyone who cannot review the diff should skip it. High-volume pipelines should skip it.

Use it for disposable experiments. Use it for recovery when the alternative is manual repair. Use it when a human checks the output.

Limitations

Fifty fixtures is a small sample. Five per class is thin. One run on one day proves nothing permanent.

Model behavior drifts. Quotas change. The free server I used today may not be the same free server next month.

temperature: 0 reduces variance. It does not eliminate it. Your numbers will differ from mine. That is the point of the harness. It gives you your numbers, not my numbers.

The takeaway

Free endpoints are excellent for disposable experiments. They are not a data-integrity layer.

Run the harness before you trust one. Then run it again next month. The harness is the artifact, not the answer.

If you want to try it yourself, MonkeyCode's free tier is a low-risk place to start. Bring your own fixtures. Bring your own skepticism.

What fixture class should I add next? I want 200 samples before I wire this into anything real.