Pin a Characterization Suite Before You Accept the First Refactor Diff

A four-year-old invoice aggregator often lives in one nine-hundred-line module with almost no tests. A teammate pastes that file into a coding agent and asks the model to clean the structure before Friday's export. The agent returns a confident diff that extracts helpers, reorders branches, and quietly changes a rounding path near tax-exempt rows. Reviewers see smaller functions and miss that weekly totals no longer match last quarter's frozen finance CSV.

That failure is not primarily an intelligence problem in the model or the reviewer. It is a missing characterization problem sitting in front of the first refactor commit. Observed behavior was never pinned against real fixtures, so every rename looked like progress until payroll compared two files. This article walks through a workflow that records current behavior first, then accepts only the smallest safe change that still satisfies those recordings.

Current developer discussion keeps returning to agents that write most of a diff. The engineering question underneath those threads is narrower and older. Which observable contracts must remain identical after someone, human or model, touches a tangled module?

Why "clean it up" destroys accidental contracts

Legacy modules accumulate three kinds of knowledge that comments almost never capture in a reviewable form.

• Branch order that accidentally encodes precedence between discounts, credits, and tax flags.
• Implicit rounding, string coercion, and locale formatting at the CSV and money edges.
• Side effects that fire only for certain customer flags, backfill dates, or retry counts.

An agent trained to reduce complexity will happily collapse those accidents into a tidier control flow. Collapse can be the correct long-term design when the accidents were never business rules. Collapse can also move a one-cent discrepancy into a different fiscal week without failing a linter.

Vibe-led refactoring is popular because the resulting file is easier to read in a pull request. The missing artifact is a characterization suite that fails when observed outputs move, even if cyclomatic complexity improved. Until that suite exists, a smaller function is not evidence of a safer change.

Step 1 — Wrap the current entry point before anyone edits it

Do not start by asking a model to rewrite the file in place. Start by wrapping the current public entry point and recording outputs from fixtures you already possess. Treat the recordings as a proposal until they run twice with identical bytes on the same revision.

# characterize_invoice.py
# Labeled example: unexecuted against your ledger; adapt paths and types.
from __future__ import annotations

import json
from pathlib import Path
from typing import Any, Callable

GOLDEN_DIR = Path("tests/golden/invoice_aggregator")


def dump_canonical(value: Any) -> str:
    return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)


def record_case(name: str, entry: Callable[..., Any], **kwargs: Any) -> Path:
    GOLDEN_DIR.mkdir(parents=True, exist_ok=True)
    payload = entry(**kwargs)
    target = GOLDEN_DIR / f"{name}.json"
    target.write_text(dump_canonical(payload) + "\n", encoding="utf-8")
    return target


def assert_unchanged(name: str, entry: Callable[..., Any], **kwargs: Any) -> None:
    target = GOLDEN_DIR / f"{name}.json"
    if not target.exists():
        raise FileNotFoundError(f"missing golden file for {name}; record it first")
    observed = dump_canonical(entry(**kwargs)) + "\n"
    expected = target.read_text(encoding="utf-8")
    if observed != expected:
        raise AssertionError(f"characterization drift in {name}")

Record from production-shaped fixtures, not from synthetic rows invented to make the function look pure. Include at least one month-end file, one tax-exempt customer, one zero-total refund, and one retry duplicate. If the module writes a file or a queue message, capture that artifact in the same canonical dump instead of asserting only the return value.

# Record once against the untouched revision.
python -c "from characterize_invoice import record_case; from invoice_agg import run; record_case('fy25_w36', run, path='fixtures/fy25_w36.csv')"

# Confirm the recording is stable before any refactor branch exists.
python -c "from characterize_invoice import assert_unchanged; from invoice_agg import run; assert_unchanged('fy25_w36', run, path='fixtures/fy25_w36.csv')"

If the second command fails on an untouched revision, you do not yet have a characterization suite. You have a flaky exporter, a clock dependency, or an unordered dict that still needs pinning. Fix recording stability before you discuss helpers, types, or agent-generated patches.

Step 2 — Promote recordings into tests that own the merge gate

A golden file that nobody runs is a souvenir. Promote each recording into a test that CI, a laptop, and a free remote shell can execute with the same command. Keep the test body boring so failures point at behavior, not at framework cleverness.

# tests/test_characterize_invoice.py
# Labeled example: add cases only from fixtures you can legally store.
import pytest
from characterize_invoice import assert_unchanged
from invoice_agg import run

CASES = [
    "fy25_w36",
    "tax_exempt_retail",
    "zero_total_refund",
    "duplicate_retry_same_day",
]

@pytest.mark.parametrize("name", CASES)
def test_aggregator_matches_pinned_behavior(name: str) -> None:
    fixture = f"fixtures/{name}.csv"
    assert_unchanged(name, run, path=fixture)

python -m pytest tests/test_characterize_invoice.py -q
git add tests/golden tests/test_characterize_invoice.py characterize_invoice.py
git commit -m "test: pin invoice aggregator characterization before refactor"

That commit should contain no production edit. Reviewers then have a baseline that is older than the cleanup story. Any later diff that claims to be behavior-preserving must keep this command green without rewriting golden files in the same change.

Step 3 — Decide what counts as the smallest safe change

Characterization makes large rewrites detectable. It does not by itself stop a model from proposing a two-hundred-line aesthetic restyle. Define a budget before anyone opens the agent chat, and reject patches that exceed the budget even when tests stay green.

The table is a process artifact, not a style preference. Teams that skip the budget still get green characterization tests after a rewrite that no single reviewer can hold in working memory. Smallest-safe-change means one behavior-preserving mechanical edit, not one pull request that happens to compile.

A useful local check is a diff budget measured after the characterization commit.

# Proposal: fail the branch if the production hunk is larger than one function extract.
git diff main --stat -- invoice_agg.py
git diff main -- invoice_agg.py | wc -l

If invoice_agg.py grew a new abstraction layer, a new config object, and a new error hierarchy, the change is no longer the smallest safe one. Put the extra design on a backlog that starts from green goldens, not from a single agent transcript.

Step 4 — Ask a model only for the budgeted hunk

Once the suite is green on the baseline revision, a coding model becomes a patch proposer instead of an unsupervised editor. The prompt should include the characterization command, the diff budget, and the forbidden change classes from the table. It should not include a vague request to make the module clean, modern, or more functional.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is useful here because the task is a constrained hunk, not a full rewrite contest. The free server option is useful because the characterization suite should run on a second machine that does not share your laptop's hidden environment variables, working directory dirt, or cached bytecode.

A practical loop looks like the following sequence, and it stays useful if you swap the editor for any other assistant.

1. Check out the characterization commit and confirm pytest is green twice.
2. Ask the model for one extract or rename that does not alter golden bytes.
3. Apply the hunk on a branch that cannot rewrite files under tests/golden/.
4. Run the same pytest command locally and on the free server checkout.
5. Reject the branch if either environment drifts or the production diff exceeds the budget.

# On the remote checkout, use a clean tree and the same fixtures.
git status --porcelain
python -m pytest tests/test_characterize_invoice.py -q --maxfail=1

Do not paste secrets into that remote shell, and do not treat a green remote run as product approval. Treat it as independent evidence that the golden files are not an accident of one developer's machine. If local passes and remote fails, you likely recorded time zones, absolute paths, or hash seed order instead of business output.

Step 5 — Keep golden updates on a separate intent track

Some teams discover during characterization that the current module is wrong. That discovery is valuable and it is also a different change. A refactor PR that "fixes" rounding while extracting a helper mixes intent until nobody can say which assertion still matters.

Split the work:

• Track A freezes today's bytes, including known one-cent oddities, under characterization tests.
• Track B writes an explicit intent test for the corrected rounding, with a product owner named in the ticket.
• Track C, only after B merges, regenerates golden files in a commit that contains no structural rewrite.

Agents collapse A, B, and C because the combined diff looks decisive. Reviewers should uncollapse them. Characterization is a lock on the past. Intent tests are a claim about the future. Refactor diffs should not carry both stories at once.

Limitations of this workflow

Characterization tests freeze accidents as well as rules. If the module formats money with banker's rounding on Tuesdays and truncation on retries, the suite will protect that contradiction. That is correct for a first refactor and dangerous if the team later confuses green goldens with a specified domain model.

The suite is only as honest as the fixtures. Internal CSV samples miss the partner file that uses a BOM, a blank trailer line, or a duplicated invoice id. If legal or privacy rules block production-shaped fixtures, this method under-approximates and should not be sold as full coverage.

Canonical JSON dumps also hide some faults. They will not catch performance regressions, lock order, or extra network calls unless those effects are included in the recorded payload. Teams that need those properties should add a side-channel log as a separate, later pin, not as scope creep in the first PR.

Free model drafts can still smuggle behavior changes inside an extract that looks mechanical. The budget and the golden gate reduce that risk. They do not remove the need for a human to read the hunk against the decision table.

Who should not use this approach

Do not use characterization-first refactoring when the current behavior is an incident in progress and the correct output is already known. Write the intent test and the fix; do not spend a day gilding the broken baseline. Do not use it on modules whose outputs cannot be stored, even redacted, without leaking customer data.

Do not use a free remote server as the only copy of fixtures that contain production extracts. Do not use the smallest-safe-change budget as a reason to postpone deleting dead code that has no callers and no goldens. Dead code with no characterization is a deletion candidate, not a refactor candidate.

Skip this workflow if the file is already covered by stable, high-signal unit tests that encode rounding, order, and side effects. In that case the characterization layer is duplicate evidence. Add it only when those tests do not exist or have been asserting mocks instead of bytes.

The smallest safe change after a green characterization suite is usually unglamorous. It is also the first refactor a later agent can extend without inventing a new weekly total. If you want a constrained place to practice that loop with free model access and a free server checkout, MonkeyCode is one option to try on a non-secret module that already has fixtures you can pin.