Read-Only Test Checklist Generation: A Free-Model Workflow That Fails Closed

Teams spend a lot of time choosing what to test after a change. Automated suites answer one question—does the current code satisfy these assertions?—but a manual smoke checklist needs a different input: given this code change, what is cheap to verify by hand before shipping? If you rebuild that list from memory, you miss changes. If you ask a code generator to write or fix tests, you hand over too much authority. A narrow middle path works better: ask a small model to draft a checkbox list from a diff summary, then validate the JSON before anyone reads it.

The workflow below assumes two availability points rather than a specific benchmark, model name, quota, or performance promise: free model access and a free server option. Those assumptions come from MonkeyCode's product outreach. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the tool does and does not do

The script is deliberately read-only:

• It reads only git diff --stat, not source code.
• It sends that summary as plain text to a model endpoint.
• It requires a structured JSON array, not free-form advice.
• It fails closed if the model returns malformed JSON.
• It prints a Markdown checklist instead of running tests or modifying files.

Keeping the input to a diff summary limits two problems at once. First, the model sees file paths and line counts, not private code; this matters when you use a free endpoint with an open data-handling policy. Second, the prompt surface stays small, so the model is less likely to drift into writing code or inventing test commands.

The artifact

This is a runnable Python 3 script except for model_call(), which you wire to the free model client you already use. The validation layer is the part worth copying.

#!/usr/bin/env python3
'''Draft a manual smoke-test checklist from a recent diff summary.

Read-only by design: this prints a checklist and does not run tests or
modify files. Replace model_call() with your free model client.
'''
from __future__ import annotations

import json
import subprocess
import sys

PROMPT = '''You are converting a code diff summary into a manual smoke-test checklist.
Rules:
- Return only a JSON array of objects with keys 'area', 'check', 'why'.
- 'area' is one of: auth, data, config, dependency, api, ui, ops, other.
- 'check' is one manual action a developer can complete in under 2 minutes.
- 'why' is one sentence tied to the diff.
Do not invent files not present in the diff.
Do not include automated test commands.
'''

def git_diff_summary(repo: str, base_ref: str = 'HEAD~1') -> str:
    return subprocess.run(
        ['git', '-C', repo, 'diff', '--stat', base_ref],
        check=True, capture_output=True, text=True,
    ).stdout

def model_call(prompt: str, text: str) -> str:
    # Wire this to your free model endpoint or CLI.
    # Keep it read-only and pass no credentials or secrets.
    raise NotImplementedError('replace with your free model client')

def parse_checklist(raw: str) -> list[dict]:
    start = raw.find('[')
    end = raw.rfind(']') + 1
    if start == -1 or end == 0:
        raise ValueError('no JSON array found')
    data = json.loads(raw[start:end])
    required = {'area', 'check', 'why'}
    if not isinstance(data, list):
        raise ValueError('payload is not a list')
    for item in data:
        if not required.issubset(item):
            raise ValueError(f'missing keys in {item!r}')
    return data

def main() -> int:
    if len(sys.argv) != 2:
        print('usage: gen_checklist.py /path/to/repo', file=sys.stderr)
        return 2
    repo = sys.argv[1]
    diff = git_diff_summary(repo)
    raw = model_call(PROMPT, diff)
    try:
        checklist = parse_checklist(raw)
    except (ValueError, json.JSONDecodeError) as exc:
        print(f'checklist generation failed closed: {exc}', file=sys.stderr)
        return 1
    for i, item in enumerate(checklist, 1):
        area = item['area']
        check = item['check']
        why = item['why']
        print(f'- [ ] {i}. [{area}] {check} — {why}')
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

Notice that parse_checklist() does not trust the model. It extracts the first [...] block, parses it, and verifies each item has the required keys. If the model adds an extra key, that is acceptable; if it omits check, the script exits non-zero and prints nothing useful. That fail-closed behavior is what makes the workflow safe to run from a terminal or a pipeline.

Example input and output

For a diff like:

 .gitlab-ci.yml        | 12 ++++++------
 src/auth/session.ts   |  3 ++-
 src/api/client.ts     | 15 +++++++++++++--
 package.json          |  2 +-

An example of the kind of checklist the script should produce, not a captured model output:

• [ ] 1. [auth] Attempt a fresh login and confirm the session cookie flags are set — session.ts changed cookie handling.
• [ ] 2. [api] Re-run the same call with a stale token and confirm the retry path — client.ts added retry handling.
• [ ] 3. [config] Run one pipeline on a branch with the new cache key — .gitlab-ci.yml changed cache settings.
• [ ] 4. [dependency] Install from a clean lockfile and run the existing smoke suite — package.json changed a dependency range.

The value is that the list is derived from the diff instead of memory. It still needs human review, but the review is faster than writing the list from scratch.

Where the free server fits

Because the script is stateless and its only output is text, it does not need a database, persistent storage, or a background worker. A small free server is enough to keep the script available to a few teammates over HTTP, provided the endpoint is read-only and does not expose repository contents. If you already have free model access and a free server option through MonkeyCode, this script maps directly onto that setup; if you use another provider, you swap only model_call() and keep the validation layer unchanged.

Do not give the server a checkout of the full repository. Pass only the output of git diff --stat into the request, or redact file paths before sending when the repo is private. If you need auditability, log only the diff summary and the validated checklist, never raw model output that may contain runtime-specific text.

When not to use this

This workflow occupies a narrow position between rigid automation and manual recall. It is not a substitute for a code review, and it is not a test runner. Keep these limits in mind:

• Private monorepos with sensitive file names: redact paths before sending anything to a free model.
• Regulated release checklists: human owners should own the final words; use the model only for a first draft.
• Automated test generation: do not let the model output drive test commands or CI jobs. The script intentionally stops at Markdown checkboxes.
• Poor JSON reliability: if the free endpoint frequently returns prose around the array, tune the parser rather than relaxing the schema.

The decision table summarizes the main trade-off:

Keep the first version small

The strongest version of this tool is the one that refuses to guess. Wire model_call(), run it against two or three merge requests, and compare the generated checklist with the one a reviewer would write manually. If the model misses an area, add a rule to the prompt. If it invents files, go back to diff summary only. If the JSON parser fails, fail the run instead of printing partial output.

Try one public repo or one private MR where the file paths are already non-sensitive. Keep the model read-only, keep the parser fail-closed, and expand only after you have concrete examples of the checklist being useful rather than merely plausible.