Measure a Free Coding Model's Patch Footprint Before You Merge It

Most free coding model evaluations stop at pass/fail. The output either compiles or it does not. That misses a more expensive failure: a patch that fixes the reported bug while quietly changing one extra function. I track three numbers before letting any model patch a repo: files touched, out-of-scope functions, and regression tests added.

A patch that violates the second or third row is not necessarily broken, but it is not a narrow fix. It needs human review before it gets write access.

I arrived at this boundary after looking at a free model and free server offer from MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server reduce the cost of running many experiments. They do not reduce the cost of an unexpected side effect in production, so the harness below is what I use before trust.

Why pass/fail is not enough

A model can produce a patch that passes the project's existing tests while changing a helper used by three other callers. If the test suite does not cover those callers, CI stays green and the regression ships. The cheaper the compute, the more candidate patches a team is tempted to generate, which makes manual review of every diff harder.

The useful metric is patch footprint: how far beyond the requested symbol the change reaches.

A runnable patch-footprint check

This script expects a Python file and the name of the function that was supposed to change. It compares the git diff against the AST line ranges for every function in the file, then reports functions that fall outside the requested target.

# patch_footprint.py
import ast
import subprocess
import sys
from pathlib import Path

def changed_line_numbers(file_path: str) -> set[int]:
    diff = subprocess.run(
        ["git", "diff", "--unified=0", "--", file_path],
        capture_output=True,
        text=True,
        check=True,
    ).stdout
    changed: set[int] = set()
    current_new = None
    for line in diff.splitlines():
        if line.startswith("@@"):
            # format: @@ -old,count +new,count @@
            new_part = line.split("+", 1)[1].split(" ", 1)[0]
            current_new = int(new_part.split(",")[0])
        elif line.startswith("+") and not line.startswith("+++"):
            if current_new is not None:
                changed.add(current_new)
                current_new += 1
        elif line.startswith("-") or line.startswith(" "):
            if current_new is not None:
                current_new += 1
    return changed

def function_ranges(file_path: str) -> dict[str, range]:
    tree = ast.parse(Path(file_path).read_text())
    out = {}
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            out[node.name] = range(node.lineno, node.end_lineno + 1)
    return out

def main() -> int:
    if len(sys.argv) != 3:
        print("usage: python patch_footprint.py <file> <target_function>")
        return 2
    file_path, target = sys.argv[1], sys.argv[2]
    try:
        ranges = function_ranges(file_path)
    except SyntaxError as exc:
        print(f"cannot parse {file_path}: {exc}")
        return 2
    if target not in ranges:
        print(f"target function {target!r} not found in {file_path}")
        return 2
    changed = changed_line_numbers(file_path)
    touched = {name for name, rng in ranges.items() if changed & set(rng)}
    out_of_scope = touched - {target}
    print("files_touched: 1")  # this check only understands one file at a time
    print(f"changed_functions: {sorted(touched)!r}")
    print(f"out_of_scope: {sorted(out_of_scope)!r}")
    print(f"verdict: {'UNSAFE' if out_of_scope else 'NARROW'}")
    return 1 if out_of_scope else 0

if __name__ == "__main__":
    raise SystemExit(main())

Keep this as a local check, not a security boundary. It only sees Python functions, it only looks at one file at a time, and it cannot judge whether a changed line inside the target function now has a different meaning.

What the output looks like

For a patch that touches process_order and accidentally also edits send_receipt, the check produces:

files_touched: 1
changed_functions: ['process_order', 'send_receipt']
out_of_scope: ['send_receipt']
verdict: UNSAFE

That is the data point I want before a human reviews the diff. The review can focus on the out-of-scope function instead of re-reading every line.

To run this across many candidate patches, I keep a small batch loop:

for patch in patches/*.patch; do
  git apply --check "$patch" || continue
  git apply "$patch"
  python patch_footprint.py app.py process_order > "results/$(basename "$patch").txt"
  git reset --hard -q
done

The loop is intentionally boring: apply, measure, reset. It treats each generated patch as disposable until the footprint check and the test suite both pass.

When this is enough

The check is not a replacement for code review. It is a filter that moves ambiguous patches into the review queue and lets narrow patches proceed to testing.

Limitations

The harness misses semantic regressions that stay inside the target function, changes in binary or generated files, deleted tests, and any patch that only changes behavior through data, environment variables, or configuration. It also does not detect a model that intentionally games the metric by naming an out-of-scope edit as a helper of the target. Treat it as one signal among several.

If your project already has a mature review workflow and a high-coverage test suite, this script may add little. If your team is generating many candidate patches because the compute is free, the footprint check becomes more valuable: cheap generation increases the raw number of patches that need triage.

If you are comparing free models, add patch footprint to your evaluation log rather than comparing pass/fail alone. A free model is useful only when you can measure the cost of its mistakes.