Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Prompt caching is one of the few cost levers that's close to free — you don't change what the model does, only what you pay for tokens it's already seen. On a stable system prompt plus tool schemas plus a big retrieved-document block, a cache hit runs about a tenth the price of a cache miss on those tokens. The catch is that "cache hit" is an exact-prefix match with a five-minute clock on it, and nothing in the response screams at you when you've broken it. You just pay full price, silently, forever, until you go looking.
This post covers what caching actually matches on, the five ways I've seen agents lose the discount without anyone noticing, and the two fields in the response you should be logging so a silent miss becomes a loud one.
What actually gets cached
Anthropic's prompt caching (and the equivalent on other providers) works on a prefix, not on arbitrary reused chunks. You mark up to four points in your request with a cache breakpoint, and the API caches everything from the start of the request up to each marked point:
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
system=[
{
"type": "text",
"text": SYSTEM_PROMPT, # large, stable
"cache_control": {"type": "ephemeral"},
}
],
tools=[
{**tool_def, "cache_control": {"type": "ephemeral"}}
for tool_def in TOOL_SCHEMAS # last tool gets the breakpoint
][-1:] and TOOL_SCHEMAS[:-1] + [
{**TOOL_SCHEMAS[-1], "cache_control": {"type": "ephemeral"}}
],
messages=conversation_so_far,
)
On the first call, everything up to and including a breakpoint is written to the cache — you pay a write premium (1.25× the base input price for the default 5-minute TTL, 2× for the optional 1-hour TTL) for those tokens. On every subsequent call within the TTL, if the prefix up to that breakpoint is byte-identical to what's cached, you pay 0.1× the base input price for it instead of 1×. That's the whole mechanism: write once at a markup, read many times at a 90% discount, as long as the prefix hasn't changed and the clock hasn't run out.
The word doing all the work there is byte-identical. Not "semantically the same," not "functionally equivalent" — identical tokens, in identical order, from the start of the request. This is where silent misses come from: the prefix looks the same to you, and isn't, to the cache.
Five ways agents lose the discount without noticing
1. Volatile content ahead of the breakpoint. The most common one. Someone puts f"Current date: {datetime.now()}" or a request ID or a live token count at the top of the system prompt, ahead of the cache breakpoint, because it feels like it belongs with the other system-level facts. Every call now has a different prefix from the first byte, so every call is a full write, never a read. This is a direct lesson from the context window as a cache framing: the things you're trying to keep stable are the ones that drive your caching strategy. The fix is mechanical: anything that changes call-to-call goes after the last cache breakpoint, never before it.
# Breaks caching on every single call:
system_text = f"Current date: {datetime.now().isoformat()}\n\n{STATIC_INSTRUCTIONS}"
# Cacheable — volatile bit moved after the breakpoint, into the first user turn:
system_text = STATIC_INSTRUCTIONS # cache_control goes here
first_user_message = f"[Current date: {date.today()}]\n\n{actual_request}"
2. Non-deterministic serialization of tools or schemas. If your tool definitions are built from a dict and your language's dict-to-JSON serialization isn't order-stable (or you're merging tool sets from multiple sources in a different order per call), the content of your tools list is unchanged but its serialized bytes aren't. The API hashes bytes, not meaning. Build your tool list with a fixed, explicit order and serialize it the same way every time — don't rely on insertion order from a dynamic registry lookup.
3. TTL expiry on slow loops. The default cache lifetime is 5 minutes from last use — every hit refreshes it, but a miss doesn't. If your agent's steps involve slow external I/O (a long-running tool call, a human-in-the-loop pause, a queued job it's polling), the gap between steps can exceed 5 minutes even though the prefix would otherwise still match. You get a full write every step, i.e. you're paying the write premium repeatedly and reading it back zero times. If your loop has slow steps, use the 1-hour TTL ({"type": "ephemeral", "ttl": "1h"}) on the breakpoints that cover genuinely stable content — the 2× write cost amortizes over far more reads if your step interval is minutes, not seconds.
4. The breakpoint placed after the part that varies. This is the mirror image of #1, and it's the one people get backwards when they're trying to be thorough: putting the cache breakpoint at the end of the request, after the per-call user content, on the theory that "more coverage is better." It isn't — everything up to a breakpoint has to be identical for that breakpoint to hit, so a breakpoint placed after volatile content never hits, and you've paid the write premium on the stable part for nothing. Put breakpoints immediately after the last byte of what's actually stable: end of system prompt, end of tool definitions, end of a large retrieved-document block — not at the end of the whole request.
5. Caching the wrong layer entirely. This is the one I wrote about from the other direction in the context window is a cache, not a memory: the part of an agent's context that changes fastest is the running transcript — every tool call and result appended as the loop continues — and that's exactly the part prompt caching helps least, because each new turn moves the boundary of what's identical to the previous call. Caching rewards a stable prefix; an append-only transcript is the least stable thing in the request. The opposite move — actively compacting early history to free space — breaks your cache even more, since the prefix is no longer byte-identical. If you're mutating or reordering history for context-window management, you're already trading away cache hits on that region — that's a real cost, not a free lint fix, so decide per-region which one you need more.
How to tell if you're actually getting hits
The response (and the streaming message_start/message_delta events) includes a usage object with cache_creation_input_tokens and cache_read_input_tokens alongside the regular input_tokens. These are the ground truth — not a log line you wrote, not an assumption from your code structure, the actual accounting from the call that happened.
usage = response.usage
write, read, base = (
usage.cache_creation_input_tokens,
usage.cache_read_input_tokens,
usage.input_tokens,
)
total_tokens = write + read + base
hit_rate = read / total_tokens if total_tokens else 0.0
print(f"cache hit rate: {hit_rate:.0%} (write={write} read={read} base={base})")
Log this per call, not just per session — a healthy agent loop should show cache_creation_input_tokens on roughly the first call after a cold start or TTL lapse, and cache_read_input_tokens carrying the bulk of the stable prefix on every call after. If cache_read_input_tokens is consistently zero across a run where you expect a stable prefix, one of the five things above is happening, and the fix is to bisect the request: strip it down to just the system prompt and tools with nothing else, confirm that alone gets hits, then add pieces back until the hit rate drops.
The arithmetic that makes this worth doing
Take an agent with a 6,000-token system prompt plus tool schemas (stable across a run) and a task that takes 15 steps, each step appending roughly 400 tokens of transcript. Sonnet-class base input pricing is $3/MTok; cache writes run 1.25× that ($3.75/MTok) for the 5-minute TTL, cache reads run 0.1× ($0.30/MTok).
Without caching: each of the 15 calls resends the full growing transcript plus the 6,000-token stable block. Total input tokens ≈ 15 × 6,000 + 400 × (0+1+…+14) = 90,000 + 42,000 = 132,000 tokens, all at $3/MTok → $0.396 for that one run's input tokens.
With caching, assuming the 6,000-token block hits its breakpoint correctly and the run stays inside the TTL: call 1 writes 6,000 tokens ($3.75/MTok → $0.0225), calls 2–15 read that same 6,000 tokens at $0.30/MTok (14 × 6,000 × $0.30/1e6 = $0.0252) instead of paying base rate for it 14 more times. The transcript portion (42,000 tokens across the run) still isn't cacheable — it's the volatile part — so it's still billed at base rate: $0.126. Total: $0.0225 + $0.0252 + $0.126 ≈ $0.174, a 56% reduction on this run, and the reduction gets better as the stable block grows relative to the transcript, which is the common case for agents with large system prompts or big retrieved-context blocks.
That's the shape of the win: caching doesn't touch the part of your cost that's growing (the transcript, still O(N²) across a long run — see long agent runs are quadratic), it discounts the part that's fixed. Both levers matter; they're not substitutes for each other.
What I'd do
Put your genuinely static content — system prompt, tool schemas, any large fixed reference doc — first in the request, with a cache breakpoint immediately after the last byte of it, and nothing volatile ahead of that boundary. Log cache_read_input_tokens and cache_creation_input_tokens on every call from day one, not just when you suspect a problem — a hit-rate graph that quietly drops to zero is the only way you'll catch a serialization change or a stray timestamp before it costs you a month of silent full-price calls. And if your loop has slow steps, check the TTL against your actual step interval before you assume the 5-minute default is fine.
