An AI coding agent dismantled a core architectural assumption across 189 files of a 717,725-line production application, and no human read the generated code for correctness. A case study posted to arXiv on August 12, 2026 documents the run in unusual detail: 14 cycles auditing a formal specification against the source, 17 cycles auditing the code against that frozen specification, 201 defects corrected across 31 audit passes before anyone executed the program, three days elapsed, and $2,430 spent.
Key facts
- The codebase is a 717,725-line production TypeScript application across 3,648 files; the change touched 189 files, and the two commits together total 288 files, 34,770 insertions and 16,422 deletions.
- 201 defects, ambiguities and architectural deviations were corrected across 31 audit passes before a human ran the program.
- Elapsed time: three days. Cost: $2,430.
- Submitted August 12, 2026 as arXiv:2608.12440; the full specification and over 1,500 pages of raw session logs are published as evidence.
The task is the part that makes this interesting, because it is the kind of change engineers normally refuse. The application had a core lifetime invariant baked through it: a UI panel stays open for the duration of an AI request. Everything downstream assumed it. The goal was to remove that guarantee, so that a streaming generation survives its panel being closed, can be reattached to the same live stream when the panel reopens, and shows a detached stop control while hidden -- with no loss and no duplication of output. The author assessed the change as effectively infeasible through incremental refactoring, the sort of thing that conventionally justifies a rewrite instead.
What replaced human review was a loop with a stopping rule. The agent first wrote a formal specification of the target behavior, then spent 14 refinement cycles checking that specification against the actual source code, fixing the spec every time reality disagreed with it. Once the spec stopped generating findings it was frozen. The agent implemented against the frozen spec, ran a separate compile-and-test feedback loop, and then spent 17 verification cycles auditing the resulting code against the same frozen spec. The convergence criterion was empirical: two consecutive verification passes returning zero findings.
The useful analogy is a building inspection where the inspector and the builder are the same person, but the blueprints are locked in a safe first. The trick is not that the agent is trustworthy. It is that the specification is frozen before implementation begins, so the agent cannot quietly move the target to match what it built. Everything after the freeze is a comparison against a fixed document, which is a much easier thing for a model to do reliably than "write correct code."
Two claims in the title deserve deflating. "No test oracle" does not mean no tests -- it means there was no pre-existing test that could tell you whether the new lifetime behavior was correct, because the new behavior had never existed. The paper says compile, type and unit-test failures were handled in a separate feedback loop and that the pre-existing unit suite ran after the change. Likewise, "no human code review" means nobody inspected the generated code for correctness inside the loop; the author still confirmed the implementation across three segments and eventually ran the software. The honest version of the claim is that a human never read the diff, which is startling enough on its own for a 34,770-line insertion.
What broke was ordinary large-refactor pain rather than anything exotic. The refinement phase kept surfacing hidden dependencies and expanding scope as it went. At one point the agent refused to perform a partial migration on the grounds that it would leave the repository in an inconsistent state -- a judgment most engineers would endorse. One interface decision was revised after release and needed a one-file fix plus tests. Across the first session and roughly thirty later ones, the author reports the software behaved as specified with no bug observed.
The counter-argument is the one the author makes himself, at length. This is one case, on one closed-source codebase, with one model, self-reported by the person who designed the protocol, with no control condition and no comparison against a human team doing the same refactor. There is no proof that the convergence criterion -- two clean passes -- correlates with correctness rather than with the auditor running out of ideas. The top question on the paper's Hugging Face discussion page challenges exactly that, and the author's reply concedes there is no mathematical proof, only manual stress testing plus the published logs. Publishing 1,500 pages of raw session transcripts is a real answer to that objection, even if it is not a complete one.
What makes the case study worth reading anyway is how closely the protocol matches what the labs themselves now describe as normal practice. Anthropic's guidance on building loops in Claude Code defines a loop as repeated cycles until a stop condition is met and recommends a second agent for code review; Addy Osmani's "loop engineering" argues for designing the loop that prompts and checks the agent instead of prompting turn by turn. This paper is what that advice looks like when someone actually instruments it and publishes the receipts. It also lands in the middle of a live argument about whether agent instruction files are accumulating rules nobody remembers the reason for -- a frozen specification is, among other things, a rule set with a documented reason attached.
Originally published on Ground Truth, where every claim is checked against the primary source.
