The item sits at the top of the feed. "Last trading day for the first US spot Bitcoin ETF to shut down." Tagged as verified fact. Three sources, all dated August 3rd.
It is August 17th.
Fourteen days between the most recent source and the event it announces. A shutdown can be postponed. A vote can be adjourned. And I had written a rule for exactly that case a week earlier, a rule that says in plain words never to copy a two-week-old announcement as if it were still true.
The setup, for anyone arriving cold
This site runs an automated watch. Every 24 hours a scheduled task wakes a Node script that calls a language model with web search access, asks it to sweep the last few hours of crypto and financial news, and demands strict JSON in return. That JSON feeds a public page: a list of items, each with a title, a summary, its sources and their publication dates.
The model does not decide alone what it brings back. It receives a prompt, meaning the instruction text sent on every call, here 8,900 characters acting as a specification: which categories to cover, which sources to favour, the expected output format, and freshness rules defining what deserves to enter the feed. The prompt also requires a certainty label on every item, from FAIT_VERIFIE when two independent primary sources agree, down to SPECULATIF for a hypothesis.
The part that matters here: this prompt is not a fixed text, it is a template. It holds markers in double braces that the script fills in right before the call. {{DATE}} becomes the current timestamp, {{FREQUENCY_HOURS}} becomes the number of hours in the cycle, {{PRICES}} becomes a block of prices pulled from an API. So the text the model receives is never exactly the one I read in my editor.
That gap is where the whole story happens.
The watch pipeline: scheduled task, template markers filled in, model call with web search, strict JSON, public page. The defect sits at the fill-in step. Scheduled task, every 24 h Template: markers get filled in the defect is here Model + web search Strict JSON, one object per item Public watch page
The model never sees the template, only the text produced at the fill-in step. Whatever goes wrong there shows up nowhere else.
The rule existed, and it did nothing
Here is the relevant passage, exactly as stored in the config:
If you keep an item whose event falls in this cycle but whose sources are
ALL older than {{FREQUENCY_HOURS}}h, you may NOT publish it as is.
Check that it still holds:
- search for a RECENT source confirming the event is happening ;
- if you find one, add it to the sources and keep FAIT_VERIFIE ;
- otherwise downgrade to PROBABLE, and say in the summary that the
deadline has not been reconfirmed.
The rule offers two outcomes and the output honours neither. No recent source added, no downgrade, no caveat in the summary. The item ships as verified fact, with the confidence of something confirmed that very morning.
At this point the first reflex is always the same: the model disobeyed. Push harder, add more capitals, repeat the instruction in two places.
That reflex is what cost me the most time.
The expensive reflex: blaming the wording
I did what everyone does. I reread the prompt and found it poorly built, which it was: four separate blocks each declare themselves top priority, the freshness rule and the reconfirmation rule contradict each other on edge cases, and across 8,900 characters there is exactly one concrete example.
So I rewrote it. Section tags instead of shouty headers, a priority block that explicitly ranks the families of rules, the freshness test turned into a numbered four-branch procedure, two worked examples including one deliberately downgraded item. From 8,900 to 11,800 characters, and a far clearer read.
Then comes the only question that matters: better, but measured how?
A prompt that returns JSON has the pleasant property of being gradeable without being read. The rule above translates into an executable test almost word for word:
const times = item.sources.map(s => Date.parse(s.published_date)).filter(Number.isFinite);
const newest = Math.max(...times);
const stale = newest < now - WINDOW_H * 3_600_000;
const hedged = /reconfirm|no recent source/i.test(item.summary);
// The rule: all sources outside the window => PROBABLE + explicit caveat.
const violation = stale && (item.certainty === 'FAIT_VERIFIE' || !hedged);
Six lines, and nobody's opinion on the beauty of the prompt carries any weight anymore. A run is now a number: how many items break the rule. All that is left is to run the old prompt and the new one, and compare.
Forty minutes for zero items
The old prompt goes first. Ten minutes later the subprocess is killed on timeout. Automatic retry, ten more minutes, killed again. The new prompt takes over and does exactly the same thing. Four calls, forty minutes, zero items compared.
At that point there are two ways forward. Shrug and say the API is slow today, or go and look. "The environment" is not a diagnosis, it is the name we give to whatever we have not measured.
The good news is that this kind of call can be watched closely. The CLI's stream mode emits one JSON event per step, and timestamping each line as it comes out is enough to see where the time goes:
claude --print --output-format stream-json --verbose \
--tools 'WebSearch,WebFetch' -p "$PROMPT" \
| while IFS= read -r line; do printf '%s\t%s\n' "$(date +%s.%N)" "$line"; done > trace.jsonl
A small script then aggregates the gaps between events. Verdict:
total: 618.9s across 492 events
458.8s 74% model generation between tools
81.0s 13% tool results
70.7s 11% writing the final answer
tool calls: 20 WebSearch, 6 WebFetch
longest wait outside the final answer: 7.6s
No holes. No sixty-second pause that would betray a backoff, and every quota event says the call is allowed. The closing event gives the rest: 343 seconds of API time out of 619 seconds of wall clock, the remainder being web search latency, roughly ten seconds per call across twenty-six calls.
The job is simply long. The ceiling was ten minutes, the previous ten cycles landed between 296 and 445 seconds, and the margin looked comfortable until a busy news day ate it. The worst part is the retry design: it starts from scratch. A run killed at 599 seconds does not resume at 599, it restarts at 0 and gets killed a second time. Paying twice to fail twice.
Ceiling raised to fifteen minutes. The next cycle came in at 836 seconds, precisely the zone where the old setting guaranteed an empty feed.
The bug was one word
That leaves the original question, the ignored rule. And here is a reflex worth its weight in gold: do not reread the prompt in your editor, look at what actually left the machine. The process is running, its arguments are readable.
ps -eo pid,etimes,args | grep 'claude --print'
And in the text scrolling past, this:
A news item = a recent EVENT within the last {{FREQUENCY_HOURS}} hours.
The placeholder is still there. Not substituted. The model received, word for word, a rule with a hole where its threshold should be.
The cause is the following line, which looks perfectly innocent:
prompt = rawPrompt
.replace('{{DATE}}', new Date().toISOString())
.replace('{{FREQUENCY_HOURS}}', String(frequencyHours))
.replace('{{DEDUP_HINT}}', dedupHint);
In JavaScript, String.prototype.replace called with a string pattern replaces only the first occurrence. You need a regular expression with the global flag, or replaceAll, to reach the others. It has been in the spec forever, it throws no error, no linter warning, and it goes unnoticed as long as a template holds a single occurrence of each marker.
This prompt held four of that particular marker. One substituted, three untouched.
The template holds the same marker four times. Substitution fills only the first: the next three rules reach the model with a hole where their threshold should be. The prompt template What reaches the model Cycle mission {{FREQUENCY_HOURS}} Item freshness {{FREQUENCY_HOURS}} No recent article {{FREQUENCY_HOURS}} Deadline reconfirmation {{FREQUENCY_HOURS}} replaced untouched untouched untouched Cycle mission 24 hours Item freshness {{FREQUENCY_HOURS}} No recent article {{FREQUENCY_HOURS}} Deadline reconfirmation {{FREQUENCY_HOURS}} .replace() with a string pattern: first occurrence only
Only one of the four rules gets its threshold. The other three arrive with a hole, and a rule without a threshold filters nothing.
That detail changes the whole diagnosis. The model never disobeyed. It applied to the letter a text I had not read, because I was rereading the one in my editor rather than the one going over the wire. Pushing harder, adding capitals, repeating the instruction: none of it would have worked, because the problem was never the wording.
The same threshold lived in three places
While fixing this I ran into the cousin of the same problem. The prompt hardcoded a 48-hour freshness window. The code computed it:
const windowH = Math.max(frequencyHours * 1.5, 48);
For a 24-hour cycle both land on 48. By luck, not by construction. Change the watch frequency and the two silently diverge without anything breaking: the prompt keeps announcing a threshold the pipeline no longer applies.
And there was a third copy, in the log line that recomputed the formula on its own to display it. As a result the logs announced a 36-hour threshold while the code applied a different one. Debugging from logs that lie costs you an evening.
The fix is not clever, and that is the point. One exported function owns the formula, the code uses it, and the prompt receives it through a dedicated marker instead of copying it by hand. A constant that code and prompt must agree on needs a single owner, exactly like between two modules.
What measurement did to my rewrite
Once substitution was repaired, the comparison became possible. Three runs, the same six-line judge, the same day:
items violations duration
old prompt, as is 8 2 703 s
old prompt, fixed 9 0 836 s
my rewrite 4 0 680 s
The second row says it all. The one-word fix drops violations to zero, on the prompt I had judged badly written. My rewrite improves nothing on that criterion and returns half as many items, on the same news cycle.
The reason sits in my own text. I had added a priority block ranking volume last, plus a sentence saying five solid items beat fifteen padded ones. The model obeyed. On a daily feed where nine sourced items were perfectly legitimate, it threw five away to please me.
My prompt was better to read and worse to execute. I spent an hour writing the solution to a problem that did not exist, and the only piece worth keeping is the two worked examples: they are why the model produced a downgraded item flagged as probable with an explicit note that the deadline had not been reconfirmed. The rest goes in the bin, including the block I was proudest of.
What I take away
A prompt is code. Nobody judges a function by reading it aloud, you run it and look at what comes back. Yet put a prompt in front of people and everyone starts debating style and phrasing as if it were a cover letter.
The three moves that actually produced knowledge here are utterly mundane. Read what goes over the wire rather than what sits in the editor. Timestamp the events rather than assume things are slow. Write six lines of automated judge rather than wonder whether it is better.
And one more, the least pleasant: accept that measurement can invalidate your work. My rewrite was objectively clearer, better structured, nicer to read. It was also worse. Without the judge I would have shipped it, sincerely convinced I had improved the system.
When a model ignores a rule, the first question is not "how do I word this better". It is "did the rule arrive".
