A maintainer watched a coding agent close a ticket with a one-sentence summary. The commit was present. The diff was not. The agent said the pagination logic was fixed. The file had not changed. This is a common failure in agent-assisted development. The workflow works until it stops working. The difficult part is not writing the prompt. The difficult part is proving that the prompt changed the code.
This article describes a small reproducible diff audit. It uses a free model gateway and a free server option from MonkeyCode. The audit compares the stated intent of a task with the actual staged diff. It fails fast when the diff is empty. It asks the model to classify the patch. It writes a JSON report for a CI job to consume.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The failure mode
Coding agents produce text before they produce code. They summarize, plan, and claim. A maintainer can read the summary and believe a fix exists. The commit log can look right. The actual diff can be zero lines. This happens for several reasons. The agent may hit an internal error. The model may choose a no-op path. The tool may be denied write access. The context window may truncate the patch. The result is the same. The task is marked done without a code change.
A second failure is subtler. The diff is not empty, but it misses a required file. A fix may add a comment instead of a regression test. The agent may edit a generated file and leave the source untouched. A human reviewer may catch this in small PRs. In a large batch of agent-generated changes, it is easy to miss. A small script can catch the obvious failures. A model review can flag the ambiguous ones.
The audit workflow
- Define the task intent and required files.
- Let the agent attempt the patch.
- Stage the patch with
git add. - Run the audit script.
- Read the JSON report and exit code.
The audit does not need a paid API. It can point MODEL_ENDPOINT at MonkeyCode's free server or any OpenAI-compatible endpoint. The free model access removes the cost barrier for small projects.
The reproducible check
The script below uses only Python's standard library. It captures the staged diff. It counts added lines. It checks for required file names. It sends the diff and intent to a model endpoint. It returns a JSON report and a nonzero exit code on obvious failures.
import os
import subprocess
import json
import urllib.request
spec = {
'intent': os.environ.get('TASK_INTENT', 'Fix pagination off-by-one'),
'required_files': os.environ.get('REQUIRED_FILES', 'src/paging.py,tests/test_paging.py').split(','),
'min_added_lines': int(os.environ.get('MIN_ADDED_LINES', '2'))
}
def get_diff():
result = subprocess.run(['git', 'diff', '--staged'], capture_output=True, text=True)
if result.returncode != 0:
raise SystemExit('git diff failed')
return result.stdout
def count_added(diff):
return sum(1 for line in diff.splitlines() if line.startswith('+') and not line.startswith('+++'))
def check_required_files(diff, files):
missing = []
for f in files:
if f not in diff:
missing.append(f)
return missing
def model_classify(diff, intent):
endpoint = os.environ.get('MODEL_ENDPOINT')
if not endpoint:
return {'status': 'skipped', 'reason': 'MODEL_ENDPOINT not set'}
payload = {
'messages': [
{'role': 'system', 'content': 'A reviewer returns JSON with keys status and reason.'},
{'role': 'user', 'content': f'Intent: {intent}. Diff: {diff}. Does this diff address the intent?'}
]
}
data = json.dumps(payload).encode('utf-8')
req = urllib.request.Request(endpoint, data=data, headers={'Content-Type': 'application/json'})
with urllib.request.urlopen(req, timeout=30) as resp:
body = resp.read().decode('utf-8')
parsed = json.loads(body)
content = parsed['choices'][0]['message']['content']
return {'status': 'reviewed', 'model_text': content}
def main():
diff = get_diff()
added = count_added(diff)
missing = check_required_files(diff, spec['required_files'])
review = model_classify(diff, spec['intent'])
report = {
'empty_diff': diff.strip() == '',
'added_lines': added,
'missing_required_files': missing,
'review': review
}
print(json.dumps(report, indent=2))
if report['empty_diff']:
raise SystemExit('FAIL: staged diff is empty')
if added < spec['min_added_lines']:
raise SystemExit('FAIL: too few added lines')
if missing:
raise SystemExit(f'FAIL: missing required files: {missing}')
if __name__ == '__main__':
main()
Run it after staging the agent's changes:
export TASK_INTENT='Fix pagination off-by-one'
export REQUIRED_FILES='src/paging.py,tests/test_paging.py'
export MODEL_ENDPOINT='https://your-free-server.example/v1/chat/completions'
git add .
python diff_audit.py
The report contains four fields: empty_diff, added_lines, missing_required_files, and review. The exact model response shape depends on the gateway. The script parses the common OpenAI-compatible structure. A custom gateway may need a small adapter.
What the check proves
The check proves three things. First, a diff exists. Second, the diff touches the files the maintainer expected. Third, a model review returned a status. It does not prove correctness. The model might approve a broken patch. A passing check is a signal to continue, not a replacement for tests. A failing check is a hard stop. The empty diff and missing file checks are deterministic. The model review is probabilistic.
The workflow is most valuable in a pre-commit hook or CI pipeline. It runs fast. It writes a small report. It does not require a paid model. It can use a free server when the codebase is not sensitive.
Limitations
The audit should not be treated as a security review. It does not validate secrets, dependencies, or permission changes. The model may miss a subtle logic error. The free model endpoint may have rate limits or response constraints. The free server option may not be suitable for production workloads. The operator should verify the current limits from the provider's own documentation before relying on it.
The script also assumes the diff is staged. Unstaged changes are ignored. A developer can stage only part of the patch and get a false positive. The required file check is a simple substring match. It can fail if the file name appears only in a comment. Those limitations are acceptable for a first-pass triage tool. They are not acceptable for a final merge decision.
Who should not use this approach
A team working with proprietary code should not send that code to an external model unless the terms permit it. A team that needs deterministic CI output should rely on the empty diff and required file checks only, without the model review. A team that wants full static analysis should use a real linter and test runner. This audit is a small safety net, not a complete quality gate.
The first test should be a simulated empty diff. Stage nothing, run the script, and confirm the fail-fast path. Then stage a known-good patch and confirm the report. That two-step check takes a few minutes. It reveals whether the endpoint and adapter work before the workflow is used in a real project.
