The cheapest LLM call is the one you don't make: a caching layer that actually pays off

The cheapest LLM call is the one you don't make: a caching layer that actually pays off

In the last post I wrote about routing across providers to cut our bill ~40%. Caching was the second lever — and honestly the more underrated one. Here's what we learned shipping it.

Routing gets most of the attention because it's sexy: traffic dancing across providers, failover kicking in, dashboards lighting up. But the single biggest cost lever we pulled after routing wasn't smarter routing. It was not calling the model at all.

Why caching gets ignored

When people talk about LLM cost, they picture the per-token price. That's the wrong unit. The question is how many of your calls are genuinely new information versus repeats wearing a costume.

We were shocked at the overlap. Once we started measuring, a large share of production traffic was re-asking near-identical things:

• The same system prompt + near-identical user input, re-embedded every time.
• The same retrieval-augmented question asked by different users within minutes.
• Deterministic pre/post-processing steps recomputed on every request.

None of that needs a fresh model call. It needs a cache with a brain.

Three layers that actually paid off

1. Exact cache (the boring one that works immediately)

Hash the full request (system + messages + params). If you've seen it, return the stored completion. Obvious, but most teams skip it because "our prompts are dynamic." They usually aren't that dynamic.

import hashlib, json

def cache_key(req):
    return hashlib.sha256(json.dumps(req, sort_keys=True).encode()).hexdigest()

def complete(req):
    k = cache_key(req)
    hit = store.get(k)
    if hit:
        return hit  # zero tokens spent
    out = model_call(req)
    store.set(k, out, ttl=300)
    return out

This alone killed a chunk of bill on our highest-traffic endpoints.

2. Semantic cache (the one people underestimate)

Exact matching misses the real win: similar prompts returning similar answers. Embed the user turn, store embeddings in a vector index, and on each request check for a neighbor above a similarity threshold (we use ~0.92). If found, reuse the prior completion.

The catch: semantic caching is only safe for deterministic-ish tasks (classifications, extractions, stable Q&A). Don't cache creative generation — you'll serve stale voices. We scope it tightly and it still covers a surprising volume.

3. Deterministic-step cache

A lot of "LLM calls" are actually deterministic work wrapped in a prompt: parsing, normalization, format conversion. We moved those to pure functions computed once and reused. It's not even a model cache — it's just not pretending the model is needed.

Tuning without breaking things

• TTL by volatility. Stable reference answers: long TTL. Fast-moving data: short or none.
• Token budget for the lookup. An embedding + vector search costs tokens too. Make sure the cache check is cheaper than the miss — for us it is, by a wide margin.
• Measure hit rate, not just savings. Hit rate tells you when caching stopped helping (prompt drift, new use cases) so you can re-scope.

The numbers

• Cache hit rate across cached endpoints: ~35%.
• Additional bill reduction on top of routing: meaningful — combined with routing we're now well past the original 40% on the endpoints that use both.
• p95 latency on cached hits: sub-50ms instead of hundreds of ms. Users notice the speed more than the savings.

None of this is exotic. It's the same caching discipline people have applied to databases for decades, applied to model calls where the per-hit savings are bigger.

Where this fits with the rest

Routing moves traffic to the cheapest healthy provider (how we cut the bill with routing). A circuit breaker keeps a flaky provider from turning an outage into a bill explosion (the pattern we use). Caching is the layer underneath both: the call you skip is the call you never have to route or protect.

If you're optimizing the same thing

Getting reliable, affordable model access set up for a team has its own headaches — provider quotas, region limits, payment friction. If any of that sounds familiar, I'm happy to compare notes. Find me here or DM me; no pitch, just war stories.