Originally published on hexisteme notes.
I have a script that turns an evaluation run into a markdown report. Numbers come from f-strings. Sentences come from string literals sitting in the same function. That split is so ordinary I never looked at it — and it is the whole defect.
A literal exists independently of the data. When the data changes, the literal stays. It compiles. It renders. Nothing complains. The number two lines above it moves, and the sentence does not.
Six sentences, four failure paths
An adversarial review of my own report found six prose claims that its own data contradicted. What surprised me was that they had not failed the same way:
- Two were drift. One said "6 blocked" after the candidate set grew from 27 to 28 items — the count moved, the sentence did not. Another cited a record ID as a worked example; that ID existed in an earlier snapshot and not in the current run.
- Two were wrong from the first commit. One described a state machine backwards: it said a blocking condition meant "replanning was never even attempted," when in the code that same condition is precisely what triggers replanning. It had never been true.
- Two were unsupported assertions. One attributed an outcome to a specific earlier fix as "the direct result." The counterfactual snapshot that would establish that was never saved.
The common structure is not rot. It is that a literal has no dependency on the thing it describes. Drift is one way that hurts you. Being wrong on day one is another. Neither is detectable by reading the sentence, because the sentence always reads fine.
The prescription is not "derive everything"
My first instinct was: replace every literal with an expression that reads from the source. That is the right instinct and it does not fully work.
Of the six, three became derived expressions — three of the six now read from the same data structure that produces the number printed next to them, instead of restating it by hand.
The other three could not be derived at all. There is no stored counterfactual for "would this have happened without that fix." There is no artifact for a value observed in a previous session. Forcing derivation there would have meant inventing a source.
So the rule I actually landed on is a pair:
Derive what has a source. For what has none, write the reason it has none into the sentence itself.
The second half matters more than it looks. "This is a session note, not reproducible from this run's artifacts" is eleven words that convert an unverifiable claim into a verifiable statement about verifiability. Leave it as a bare literal and it rots on the next round. Mark it and the next reader — including future you — stops asking whether it should have been derived.
If you write this down as "we replaced all the literals with derived values," you have just written a seventh literal. I know because that is the subheading I wrote in the correction log, with a table two lines below listing the two counterexamples I had personally just written. The review caught it. That sentence is now struck through in the document, above the table that refutes it.
Then the checker passed the bug it was built for
Because this class kept recurring, a previous round added a checker: for a registry of specific claims, pull the supporting evidence out of the JSON, and if that evidence set is empty, refuse to let the corresponding paragraph state a confident conclusion. Conclusion vocabulary near an empty evidence set is a failure.
The obvious false-positive problem is hedged language. A paragraph can legitimately say "this did not fire" right next to a sentence about something that did. So the first version exempted a conclusion word if a hedge marker appeared within roughly a hundred characters.
Character distance is not clause membership.
Real sentences contrast. "A did not fire, but B did" puts the hedge and the conclusion a few words apart and pointing at different things. According to the change log — the old rule's code is gone from the repo, so I can't re-run it myself — the checker exempted the second clause because of a marker belonging to the first, and it passed exactly the sentence shape it existed to catch.
Worse: according to that same change log, the checker's own built-in tamper test said it worked. That test reverted the whole paragraph to its pre-fix text and confirmed a failure was raised. But the failure came from an unrelated sentence elsewhere in that paragraph, not from the headline under test. The test passed by coincidence, and the coincidence read as proof.
Two fixes:
- Split by clause, not by distance. Boundaries are sentence terminators plus the adversative and coordinating connectives the language actually uses to pivot. (In my case an em dash is not a boundary — in this document it almost always introduces elaboration, not contrast, so treating it as a boundary would misclassify most of the corpus. Check your own corpus rather than copying my list.)
- Mutate one sentence, not one block. The tamper test now substitutes a single headline and leaves the body honest. If reverting a whole section is what makes your test go red, your test is measuring the section, not the rule.
The same mistake, one layer up
Here is the part I did not expect.
Having fixed the rule, I extended coverage to two table rows that had gone unchecked — including one that had contained a live misattribution. Then I put the original false claim back into the real file and ran the checker.
It passed.
The reason had nothing to do with clauses. That registry entry uses a different axis: when evidence is non-empty, the check is simply "is each evidence item mentioned somewhere in the unit?" The unit was the whole table row — and that row happened to name the missing item again, several clauses later, for a completely unrelated reason. Substring containment was satisfied by an accident.
So the entry I had just added to strengthen the guard was a guard that could not fail. Fixing the checker had produced, one level up, precisely the thing the checker exists to prevent: something that looks like verification and verifies nothing.
The fix was to let a registry entry narrow its check to the clause making the claim, rather than the whole row. But the durable lesson is the ordering: I only found it because I injected the defect after the change, into the real artifact, and watched. Nothing in the passing run would have told me.
That case is now a permanent mutation test. It prints, on every run, what the result would have been without the narrowing — "no missing items, i.e. a false negative" — so the reason the narrowing exists cannot quietly detach from the code.
What I would take to another codebase
- If a document is generated, grep it for sentences that assert a fact and are not built from an expression. Every one is a candidate.
- Derive the ones with a source. For the rest, state in the sentence why no source exists. Both halves, or the second group rots.
- Count before you summarize. "All of them" in a correction log is the same defect, one meta-level up.
- A check that has never failed has not been tested. Inject the specific defect it targets and confirm red.
- Mutate minimally. Whole-block reverts pass for the wrong reason and then get cited as evidence the guard works.
- After you strengthen a guard, re-inject the original defect. The strengthening is exactly when a new blind spot gets introduced, and it arrives wearing the guard's uniform.
- Consider a freshness tie: have the prose carry a digest of the artifact it describes, and have the checker rehash the artifact. Otherwise you can regenerate one side, verify the other, and get a clean pass on a stale pair. That one bit me too.
- The same split shows up outside generated reports: a CI dashboard's hardcoded "all tests green" caption sitting next to a computed pass count, or a changelog entry that was never regenerated from the diff it describes.
The checker now catches the shapes I have actually seen. It prints, on every run, what its own result would have been without the clause-narrowing step — a standing false-negative warning — and its registry documents that eight of its nine entries still lack that narrowing. Those limits are not a disclaimer. They are the part of the report that is still a literal.
More notes at hexisteme.github.io/notes.
