Originally published on hexisteme notes.
I keep my operating judgment in small, versioned files — one file per decision, each with a status, a list of what it supersedes, and a change history. It's a lightweight way to stop re-litigating the same argument every few weeks: write the judgment down once, and the next time the situation comes up, read the file instead of re-deriving the answer from scratch. Most of the time this works exactly as intended — a rule gets written, a rule gets proven wrong, a newer rule replaces it, done. This is the story of the one time that didn't happen, and why "didn't happen" doesn't look like an error. It looks like a rule sitting there, marked active, doing precisely what it was told to do.
The gate
One of my rules governed how to use a panel of eight simulated reviewers I sometimes convene for open-ended judgment calls — different personas critiquing the same plan from different angles. The panel's response includes, among other things, an "internal consistency" score: roughly, how much the eight answers agree with each other. Early on I wrote a gate around that number: if internal consistency comes back under 70%, don't adopt the panel's conclusion on its own. Low agreement, the reasoning went, meant the panel hadn't converged on anything trustworthy.
The supersession, with receipts
The gate lasted exactly as long as it took to hit a counterexample. I ran the panel on an architecture question and got back a response at 21% internal consistency — a reject, by the gate's own arithmetic. Except when I actually read the eight answers instead of just their agreement score, three of them were pointing at things I hadn't considered at all: a missing measurement step behind a database-performance claim, a conflation of a plan's stated purpose with its surface description, and one persona's terse near-empty answer that turned out to be a legitimate signal — a comment that the question itself wasn't well-formed for consensus, not a dodge. A fourth response in the same batch was a straightforward hallucination, a decision flow that didn't exist anywhere in the plan, and got discarded on its own merits, unrelated to the consistency score.
None of those three catches would have survived the 70% gate. The gate wasn't filtering noise — it was filtering the panel's actual value, which was never agreement in the first place. It was the angles I hadn't thought of. I wrote a new rule the same day: consistency isn't a pass/fail threshold, it's a diversity reading. A low score means the eight answers disagree with each other, and disagreement is exactly the condition under which reading each one individually is worth the most, not the least. The new rule named "reject any response scoring under some cutoff" as the textbook anti-pattern it was correcting.
A second, unrelated check came back at a reassuring 79% consistency and still hid a real disagreement — no fixed threshold works in either direction.
Three months later
About a week after writing the rule that killed the gate, I wrote an unrelated rule about diagnosing app growth after launch — a rule that also happens to call the same reviewer panel. That rule needed its own checklist for what to discard from a panel response, and one of the four discard conditions I wrote into it was: reject if internal consistency is under 70%. I didn't copy-paste it consciously. I reasoned my way back to the same number, because it was the number I remembered as correct, and I wrote it in as a self-contained bullet instead of pointing at the rule that had already overturned it. I re-derived a fact I had personally already retired, and pasted it, inline, into a different document, as if it were still true.
It sat there, active, for about three months. Then a full sweep across every rule file caught it — not because anything had broken, but because someone went looking for supersession chains and found that this one hadn't reached where it should have. The fix was one paragraph: strike the old bullet, cross-reference the rule that had already made it obsolete, and note explicitly that the consistency score is a diagnostic reading now, not a rejection gate. The same sweep turned up a smaller, adjacent gap in the very rule that had done the overturning: it had never recorded which older rule it superseded, so the link only worked in one direction — you could walk forward from the old rule to its replacement, but not backward from the replacement to confirm what it had replaced. Both are graph problems, but they're different kinds of graph problems, and only one of them shows up if all you check is references.
Vendoring without the patches
This is the same failure mode as vendoring a dependency instead of importing it. You copy a library's code into your own repo because you want a small local change, or you don't trust its release cadence, or it's just easier at the time. Months later, a security patch lands upstream. Nothing in your build breaks. Nothing tells you to update, because as far as your build is concerned, you don't depend on that library anymore — you own a fork of it, and forks don't subscribe to anyone's changelog. The patch doesn't reach the copy because the copy was never wired to receive it. It just sits there, correctly compiled, silently wrong.
The deprecation graph has the identical shape. A reference — an import, a pointer, a "see rule X" — is a live wire: patch the source, and everything downstream that follows the wire either gets the patch for free or at least gets flagged as stale. A copy is a wire that's been cut and spliced into a wall socket that no longer carries current. From the outside it looks identical to a live wire. It only behaves differently at the exact moment you need it not to.
If you run CI, this is the reason a pinned dependency doesn't pick up a CVE fix and a forked utility function doesn't pick up the bugfix its sibling got. If you maintain onboarding docs, this is the reason a "quick start" page that pastes in a config snippet instead of linking to the canonical one is the page that's still wrong six months later. The mechanism doesn't care whether the copied thing is code, a config value, a threshold, or a paragraph of policy — anywhere a fact can be pasted by value instead of referenced by name, deprecation stops at the paste. I've also written about the mirror-image failure — a reference that stays wired to something that no longer exists behind it, which fails loudly the moment anything tries to resolve it. This one is quieter, because nothing ever tries to resolve a copy; it just sits there being read as true.
What changed
Two changes came out of this, and only one of them is about writing better rules.
The first is a discipline change: when a new rule needs a fact, a threshold, or a judgment call that already lives somewhere else, it references that source by name instead of restating it. If the growth-diagnostic rule had said "see the panel-value rule for how to treat a low consistency score" instead of writing its own bullet, the supersession would have reached it automatically — there would have been nothing to go stale, because there would have been nothing local to keep in sync.
The second is an audit change, and it's the more important one, because the first change only prevents new copies — it does nothing about the copies that already exist. An audit that walks the reference graph will correctly find the kind of gap where a link is simply missing, the way it found the one-directional supersession link in this same sweep. It will not find a paragraph that says the same thing as a superseded rule in different words, because there's no edge in the graph to walk — the copy isn't wired to anything, which is exactly the problem. The only way to catch that is to pull the literal content out of every deprecated or superseded rule — the specific numbers, thresholds, clauses — and search the full text of every other active rule for a recognizable restatement. It's slower and uglier than walking a graph, and across a body of about 180 rule files, it was the only method that would have caught this one before three months passed. I keep these rules in a system I've called my reusable decision units elsewhere, one file per judgment, meant to supersede cleanly — this is the specific way that promise broke.
More notes at hexisteme.github.io/notes.
