The warning landed on the only people who had done it properly
I maintain a linter that reads agent config files — SKILL.md, AGENTS.md, CLAUDE.md — and fails CI when they bake in something that only works on the author's machine. One of its rules says: if you call an external CLI, declare it, or the next person won't have it.
Declaring it means naming it in frontmatter:
requires: codex
Except that anyone with more than one dependency writes the list form, because that's what YAML is for:
requires:
- codex
- gemini
My implementation only read the first shape. So the block list — the normal way, the way you write it the moment you have two of anything — was invisible to the linter, and it warned you for an undeclared CLI that you had, in fact, declared.
Read that back slowly. Authors who ignored the dependency question entirely were never flagged, because they never wrote a requires: key at all. Authors who sat down and wrote the contract properly got a warning telling them they hadn't. The rule was inverted with respect to the thing it was trying to encourage.
I shipped that. It went out in a patch release, and I only found it because a commenter used the phrase "dependency contract" and I went to re-read my own implementation of it.
Then it happened again. Twice, in one release
Two comments on a post of mine turned into new rules. One of them, unverified-write, reports a file that changes external state — git push, npm publish, an INSERT — and never reads that state back anywhere.
Before publishing, I ran it over 586 real skill files pulled from a public registry, found two false-positive shapes in the data, fixed both, and re-measured. Fire rate 0.7%, and every hit I could check by hand was genuine. I felt good about it.
Then I handed the diff to a different model for a pre-publish read, and it produced this input in about a minute:
Never run `git push --force` from this skill.
That is a git push in a code span, in a file with no read-back anywhere. My rule flagged it as an unverified write.
AGENTS.md and CLAUDE.md are full of that sentence. Writing down "don't push without asking" is the single most common act of care in that genre of file. I had built a rule that warns you for pushing, specifically because you wrote down that you must not push.
I fixed it, published, and then noticed the same shape one layer further out. Prohibitions were now excluded, but permissions were not:
- `git push` は明示の指示があるときだけ。
- Only run `git push` when the user asks.
- `npm publish` requires approval from a maintainer.
Nobody who writes those sentences has an unverified write. They have a policy. Three releases, three variations, all pointing the same way: the warning finds the author who wrote the rule down and misses the one who never mentioned it.
Why it points that way
A text-matching rule cannot see actions. It sees mentions. And mentions of a dangerous operation are not distributed randomly across authors — they concentrate in the files of people who thought about it.
The careless author's AGENTS.md doesn't say git push anywhere. There is nothing for the rule to catch. The careful author's file says it three times: once to declare when it's allowed, once to forbid the force variant, once in the actual deploy step. Two of those three are not the thing you're detecting, and both of them are evidence of care.
So the base rate is against you. Among all the files containing the string you match on, the share written by conscientious authors is much higher than in the population — and every false positive you have is drawn from that pool. The people most likely to read your warning carefully, and most likely to uninstall you over it, are the people you are most likely to be wrong about.
Static analysis has a name for the underlying distinction — use versus mention — and my older rule already knew it. Its CLI check ignores a bare `codex` in prose and only fires on codex exec build, an actual invocation with an argument. I wrote that exclusion two releases earlier, in response to the same class of complaint, and then built a new rule without it.
The audit that found nothing here
The part I want to be honest about: running against 586 real files did not catch any of this.
It couldn't. Published, downloadable skills are written to be used; they say "run this" far more often than "never run this." The prohibition shape lives in team-internal AGENTS.md files that nobody uploads to a registry. My corpus was real data, and it was the wrong real data — biased, in exactly the direction that hid the failure.
That's worth separating out, because "test against real data" is advice I've given in writing and still believe:
- A real-data audit tells you what your rule does to the corpus you can reach.
- An adversarial read tells you what your rule does to the input someone constructs on purpose.
The second one found in one pass what 586 files had not. It cost one prompt. If your detector will run on files you can't see — and a linter always does — you need both, and you should assume the corpus is the more comfortable of the two.
What actually fixed it
Not more keywords. The fix was making the use/mention distinction structural, then checking the price:
- A write signal only counts in a code context — inside a fence, or inside a backtick span. Prose saying "after that we push to git" is not a step.
- A line carrying a prohibition (
never,do not, 禁止) isn't a step. - A line carrying a condition or a permission (
only … when,requires approval, 〜のときだけ) isn't a step outside a fence. Inside one, the lines are commands, and excluding them there would drop a real finding over an incidental comment likegit push origin main # main only.
Then the part that isn't optional: re-run the corpus and prove the exclusions didn't eat the signal. Four true positives before, the same four after, fire rate unchanged at 0.7%. An exclusion you didn't measure is just a rule you deleted with extra steps.
One more thing, since your linter runs on other people's files
The same review turned up something unrelated but worse. My check for a remote copy looked like this:
\b(?:scp|rsync)\b[^\n]*\s\S+@\S+:
Greedy fill, then a search for something@something:. On a long line containing many @ and no colon, it backtracks quadratically: 77ms at 20k characters, 312ms at 40k, and it keeps going up from there. A minified blob or a base64 payload on one line of somebody's repo is enough.
I had been thinking of the input as "config files people wrote." It isn't. It's arbitrary text from strangers, and a linter that hangs is a linter that stops a stranger's CI. The fix was to stop the filler crossing an @ so there's nothing to backtrack over; an 80,000-character line is now a test case.
The rule of thumb I'm keeping
When a detector matches on text, ask who says that text most often. If the answer is "the people being careful about it," your false positives are not evenly distributed — they're aimed. And the only reliable way to see it is to hand the rule to something that is actively trying to embarrass you, because your own corpus is made of the cases you already knew about.
The tools: carrylint is the linter above; the read-back half it deliberately doesn't attempt is genchi.
