"Your cache hit rate is low" — true, and worth $0.16

Anthropic emailed our org last week: prompt cache hit rate is low, and caching repeated content "could save up to 64% of direct API spend." The email is careful with scope — direct API only, Claude Code excluded because it manages caching itself. I went to check what that 64% was made of.

Our direct API traffic over the period: 16 calls, $0.25 total, average prompt 90 tokens. They come from traceguard's rerun harness, which replays self-contained consults as fresh single-turn calls — every prompt unique by construction. A workload like that cannot hit cache: there is no repeated prefix across calls, and 90 tokens is far below the minimum Anthropic will cache at all (1,024 tokens on Opus 4.8, the model those calls ran on; 512 on Opus 5 / Fable 5). Hit rate 0%, structurally. 64% of $0.25 is $0.16.

The metric was right. The money wasn't there.

The side the email excluded

The interesting numbers sit in the traffic the email correctly excluded. We ingest our own Claude Code session logs into traceguard's trace store (Claude Code transcripts carry no cost field, so we compute list price from usage). 71 days, 158 sessions, 58,753 API messages. This is section 1 of the command output, unedited:

Hit rate is token-weighted; costs are the input side only, at list price. The two n/a rows are deliberate — no published price for a model means no guessed money, tokens counted anyway.

Caching nobody at our org configured cuts the input side by 83.9%. With output included, the whole bill lands at 1/5.5 of the no-cache counterfactual. There is nothing left to optimize here; the remaining spend is mostly first-write premiums on genuinely new content.

The keep-alive ping idea

There's a folk optimization going around: ping your session on a timer so the cache stays warm and the next turn hits. The mechanics say it could work — the cache TTL is 5 minutes by default, every hit refreshes it for free, and a 1-hour TTL costs 2× on writes. Instead of arguing, we ran the counterfactual on our own gap data.

97.3% of our in-session gaps are under 5 minutes; they need nothing. The addressable prize is 422 gaps longer than an hour: the context rewrites right after them cost at most $1,913 over 71 days (an overestimate, since it counts genuinely new content as rewrite). Bridging those same gaps with a ping every 55 minutes would have taken 6,765 pings costing $2,009 in cache reads alone, before counting the pings' own output tokens. The strategy loses money on our data, under assumptions tilted in its favor.

The reason is boring: the 1-hour TTL removed the ping's niche. Our logs show Claude Code already routes nearly all cache writes into the 1h bucket (opus-4-8: 155.9M tokens written at 1h TTL vs 56.4M at 5m). Pinging is a hack from before extended TTL existed. If a tool sells you auto-ping in 2026, ask it for the counterfactual on your logs first.

(Superseded — the Aug 18 note below has the current numbers.)

Update, Aug 16. Reid Marlow asked in the comments for cost broken out by idle gap rather than in aggregate. I ran it, and it overturns the paragraph above. Split by bucket: the 183 gaps of 1–4h would have cost $81 in pings against at most $887 of rewrite, an 11:1 win, while the 239 gaps over 4h cost $1,928 against at most $1,026. The refusal I published is a win and a loss added together. A policy that can actually be run (ping every 55 minutes, give up after 4 hours idle) costs $316 including every ping burned on the 239 gaps that outlive the cap, against at most $887 avoided. It pays.

So the correct claim is narrower than the one this section makes: unbounded pinging loses, capped pinging probably pays. Probably, because both sides of that comparison lean pro-ping. The $887 is an upper bound, since cache_creation mixes the re-established prefix with whatever the turn genuinely added, and the $316 charges pings as pure 0.1x reads of a frozen prompt. Under that tilt a refusal is safe and an endorsement is only as wide as its margin, and this one is 2.8x rather than 11x.

I had the aggregate and stopped there, which is the same mistake the email made: a number averaged over cases that behave differently is not a decision. The per-bucket money and both verdicts are in cache_audit in the repo now and ship in the next release; 1.2.0 prints the aggregate only.

(Numbers superseded by the Aug 18 note below. The reasoning stands.)

Update, Aug 17. The numbers in the note above are superseded. Not because the direction was wrong, but because 4 hours was picked to line up with a table boundary rather than solved for. Sweeping every cap from 1h to 12h in 15-minute steps, the best single value is 10h. It beats 9h45m by $7.63. Then I measured something this section had only assumed, and that one measurement moved the same cap by $18.76. A correction larger than the distance between first and second place can reorder them, so a single number is the wrong output. The answer is a band: ping every 55 minutes, give up somewhere between 9 and 12 hours idle. Anywhere in that band costs under 10% of the optimum.

Net over the window, after the ping bill: $663.69 to $811.30. That is two runs rather than an error bar. The low end treats every gap whose model I cannot determine as a cache-invalidating switch, the high end deducts only the ones I can prove. 54 of 432 gaps have no model recorded, and nothing in the data says which way they go.

What I measured is whether the model changed across the gap. A keep-alive only holds a cache for the model it was written on, so a switch makes every ping before it worthless. This section listed that as an assumption. It is 18 of the 378 gaps where both sides are known, and the rate climbs with idle time: zero in the shortest decile, 23.1% past 24 hours. All 18 were opus-4-8 and fable-5, in one direction or the other. Not one was a switch down to something cheaper. Leaving that out was not adding noise to the cap. It was pushing the cap long.

Every number here comes from 2026-05-30 to 2026-08-16, pinned behind a --benchmark flag so the run reproduces. That flag and the sweep are on main, not in 1.2.0, and land with the rest of this in the next release. The Aug 16 note was overconfident in the same shape as the paragraph it corrected: I had a number and quoted it before asking how wide it was. Checking that note turned up three more errors, one of them a denominator error of my own. That one is a separate post.

Update, Aug 18. The window did not do what I said it would. I pinned 2026-05-30 to 2026-08-16 so the run would reproduce, and it does not. The same window over the same store went from 432 expired gaps to 439, and from 168 sessions to 174, in under a day. Ingest reads a directory of transcripts, so a file that turns up late still carries messages from inside the window. Closing the dates closes the timestamps, not the corpus. The figures above moved with it: the two ends are now $650.53 and $806.82, not $663.69 and $811.30 (argmax_reference_only.net_usd in that file — the field named recommended_cap_band is a different interval).

So I stopped pointing at a command. The run those numbers come from is committed in the repo, fingerprinted over the traces it analysed, and the tool now refuses to overwrite an entry that already exists. A file can be cited. A rerun cannot. Everything else above stands: 9 to 12 hours at a 55-minute cadence, and the two ends are two runs rather than an error bar. Shipped in 1.3.0.

Reading is not measuring, again

While writing this I checked our own SDK and found the same class of bug I keep filing against other trackers. traceguard's wrap_anthropic recorded tokens_in = usage.input_tokens. On Anthropic's API, input_tokens excludes cache reads and writes — and on our traffic, cache reads run about 1,000× larger than raw input, so the wrapper under-counted by three orders of magnitude. Every field read correctly, the quantity still wrong (the splitrail #220 pattern). Fixed in PR #39, and the fix is what makes the next part possible.

A rate without a denominator is not a finding. Percentages travel well in emails; dollars make decisions. Before acting on a metric email, multiply it by the money it applies to.

Check yours

This analysis is now one command, merged in PR #40: python -m traceguard.routing_audit.cache_audit. Point it at your ingested Claude Code transcripts or a traceguard trace store, and get the table above for your own data — per-model hit rate, actual vs no-cache cost at list price, gap distribution, and a yes/no on whether keep-alive pings would pay for you (ours printed NOT WORTH IT — see the update above for what that verdict hides, and what the next release prints instead). Ships in traceguard 1.2.0 — pip install -U traceguard. The last line of ours came out like this:

Claude Code caching already saves us 84% ($12,110.31 vs $75,044.44 list). Checked with: python -m traceguard.routing_audit.cache_audit

If you run it, send me your numbers. Especially if pings win somewhere — I'd like to see the traffic shape that makes them rational.

Li Zhuojun