Write prompts for the cache, not the reader

Modern LLM APIs cache the key-value state of a prompt prefix. Send the same opening tokens twice and the second call is billed at roughly a tenth of the input price and returns noticeably faster. This one fact reorganized how we write every high-volume prompt.

The rule

Everything stable goes first, byte-identical on every call: system rules, output schema, few-shot examples, reference tables. Everything volatile goes last: the task, the retrieved documents, the user text. The cache breaks at the first differing byte, so a single timestamp in the header poisons the whole prefix.

[ system rules          ]  stable, versioned      ─┐
[ output schema         ]  stable, versioned       ├─ cached prefix, ~91% of tokens
[ 12 few-shot examples  ]  stable, versioned      ─┘
[ retrieved context     ]  volatile ──────────────── paid in full
[ the actual task       ]  volatile

Measured on one of our extraction pipelines

Same model, same accuracy, 78% cheaper and twice as fast, for a diff that only moved paragraphs.

Operational consequences

• The prefix is versioned like an API. Editing it is a deliberate release, because every edit is a fleet-wide cache flush and a visible cost spike. We tag prompts with a version constant and roll them like schema migrations.
• Batch queues are sorted by prefix. Requests sharing a prefix run adjacently so the cache stays warm instead of being evicted between hits.
• Hit rate is a logged metric. The API reports cached token counts per call; we graph them. A silent drop in cache hit rate has caught two accidental prompt edits before the invoice did.
• Randomness is banned from the prefix. No timestamps, no request IDs, no dict-ordering roulette. Serialization is canonical, keys sorted, for the same reason your build system wants deterministic inputs.

None of this touches quality. It is pure systems hygiene applied to a resource most teams do not know they are wasting.