Semantic Cache in AI Tokenomics

Introduction

A few weeks ago I was looking at logs for a support bot that answers the same handful of questions all day long. "How do I reset my password", "how do I reset my login", "I forgot my password, help", "can't log in, need to reset it". Four different customers, four different sentences, one single question underneath all of them.

The bot had a cache. It just never hit. Every single one of those four questions triggered a full model call, because the cache was checking for an exact string match, and none of those sentences matched each other character for character.

That is the moment it clicked. Most teams think their caching problem is "we need a cache." The real problem is usually "we have a cache and it is nearly useless," because it only fires when someone happens to type the exact same words as someone before them. In practice that almost never happens. People ask the same question in a dozen different ways, and an exact match cache is blind to every single one of those variations except the one it has seen before, word for word.

The fix is not a bigger cache. It is a different kind of cache, one that checks meaning instead of spelling.

Exact match caching and why it quietly fails

A normal cache works off a hash of the input. Same input, same hash, same cached response. It is fast, it is simple, and for a huge class of problems it works great.

It falls apart the moment your input is natural language written by different humans, because natural language is never exactly the same twice. "What are your hours" and "when are you open" mean the same thing to a person and mean nothing alike to a hash function. So the cache stores the first one, and then completely misses the second, third, and fourth time someone asks the same question in slightly different words.

You can watch this happen in your own logs. Pull up a week of prompts sent to any customer facing AI feature and group them by rough topic instead of exact text. You will almost always find clusters of near identical questions that never once hit each other in cache, because none of them were spelled exactly the same.

Semantic caching, the basic idea

A semantic cache does not ask "have I seen this exact text before." It asks "have I seen something that means roughly this before."

It does that with three ingredients:

1. An embedding. This is just a list of numbers that represents the meaning of a sentence. You get this by calling an embedding model, most AI providers have one. Two sentences that mean similar things end up with similar number lists, even if the actual words are completely different.
2. A similarity score. This is a single number, between 0 and 1, that tells you how close two embeddings are. 1 means basically identical meaning, 0 means completely unrelated. This is calculated with a bit of standard math called cosine similarity, you do not need to understand the math, just that "closer to 1 means more similar."
3. A threshold. This is a number you pick, like 0.92, that decides how similar something needs to be before you trust the cached answer instead of calling the model again. Put simply: every new question gets turned into numbers, those numbers get compared against numbers from past questions, and if the closest match is similar enough, you reuse that old answer instead of paying for a new model call.

Walking through it with a real example

Say your cache already has this one entry stored:

• Question: "How do I reset my password"
• Answer: "Go to Settings, click Security, then Reset Password" Now three new questions come in:

Notice the first two questions do not share many exact words with the cached one, "reset my login" versus "reset my password" barely overlaps as text. But they mean almost the same thing, so their embeddings land close together, and the cache correctly treats them as the same question. The third question is genuinely different, so its embedding lands far away, and it correctly triggers a real model call instead of returning the wrong cached answer.

The code, one small piece at a time

First, a small helper that measures how similar two embeddings are. This is the "how close are these two meanings" part:

import numpy as np

def similarity(embedding_a, embedding_b):
    """Returns a number between 0 and 1. Closer to 1 means the two
    pieces of text mean something very similar."""
    return np.dot(embedding_a, embedding_b) / (
        np.linalg.norm(embedding_a) * np.linalg.norm(embedding_b)
    )

Next, the cache itself. It just keeps a list of everything it has answered before, and checks new questions against that list:

class SemanticCache:
    def __init__(self, similarity_threshold=0.92):
        self.threshold = similarity_threshold
        self.stored_answers = []  # list of (embedding, original_question, answer)

    def find_cached_answer(self, new_embedding):
        """Looks for a past question that means the same thing as this one.
        Returns the old answer if found, otherwise returns None."""
        best_answer = None
        best_similarity = 0

        for old_embedding, old_question, old_answer in self.stored_answers:
            score = similarity(new_embedding, old_embedding)
            if score > best_similarity:
                best_similarity = score
                best_answer = old_answer

        if best_similarity >= self.threshold:
            return best_answer
        return None  # nothing close enough, this is a genuinely new question

    def save_answer(self, embedding, question, answer):
        self.stored_answers.append((embedding, question, answer))

Finally, here is how you would actually use it when a question comes in:

def answer_question(question_text, cache, embed_fn, ask_model_fn):
    # Step 1: turn the new question into numbers
    embedding = embed_fn(question_text)

    # Step 2: check if we already answered something similar
    cached_answer = cache.find_cached_answer(embedding)
    if cached_answer is not None:
        return cached_answer  # free, no model call needed

    # Step 3: no similar question found, so ask the model for real
    answer = ask_model_fn(question_text)

    # Step 4: remember this for next time
    cache.save_answer(embedding, question_text, answer)
    return answer

That is the whole mechanism. A question comes in, it gets turned into numbers, those numbers get compared against everything answered before, and only a genuinely new question ever reaches the model. The interesting engineering work is not in this core logic, it is in tuning the threshold and handling the edge cases below so this does not go wrong.

Where this gets tricky

The threshold is doing all the work. Set it too low and you will start returning cached answers for questions that only sound similar but actually need different answers, which is a much worse failure mode than a cache miss, because now you are confidently wrong instead of just slightly slower. Set it too high and you are back to almost never hitting the cache, which defeats the purpose.

A few things that help in practice:

Scope the cache per intent, not globally. Comparing every new question against every cached entry gets slow and increases the chance of a false match as your cache grows. Bucket entries by rough category first, like "billing questions" versus "technical support questions", and only compare within the matching bucket.

Treat a cache hit as a starting point for high stakes answers, not a final answer. For low stakes questions like store hours, returning a cached answer straight away is fine. For anything involving account specific details, use the cache hit to decide "this looks like a duplicate question" and then still verify against current data before answering, rather than returning old cached content wholesale.

Expire entries. A cached answer to "what is your current pricing" from three months ago is actively harmful once pricing changes. Attach a time to live to entries, and invalidate anything tied to information that changes on its own schedule.

Log near misses. Track cases where similarity was close to your threshold but did not quite cross it. That is exactly the data you need to tune the threshold correctly instead of guessing at a number.

What this actually saves

The savings show up in exactly the places exact match caching misses. Any feature that fields the same handful of real questions phrased a dozen different ways, support bots, FAQ assistants, internal help desks, onboarding flows, all of these tend to have a small number of actual distinct questions hiding behind a much larger number of distinct phrasings. Semantic caching is what lets you collapse those phrasings back down to the handful of questions they actually represent, and pay for the model call once per question instead of once per phrasing.

Limitations and caveats

• Embeddings and similarity comparisons are not free either. There is a real cost and latency tradeoff here, generating an embedding and comparing it against a growing cache takes time and, depending on your embedding model, its own token cost. For a small number of cache entries this is trivial, for a very large cache you likely want a proper vector index instead of the naive linear scan shown here.
• A wrong cache hit is worse than a cache miss. A cache miss just costs you an extra model call. A wrong hit gives someone a confidently incorrect answer that looks exactly like a correct one. Tune conservatively and monitor for this specifically, not just for hit rate.
• This works best for questions with stable answers. Anything time sensitive, personalized, or tied to live account data needs the expiration and verification handling described above, it is not a drop in replacement for checking current data.
• I don't have search enabled while writing this, so I have not verified how widely semantic caching is already documented elsewhere. It is a known technique in retrieval and vector database circles, the angle here is applying it specifically to token cost economics rather than treating it purely as a retrieval optimization.

Wrapping up

If you already have a cache and it still feels like your AI bill is too high, the cache is probably not the problem, the matching logic underneath it is. Pull a sample of your actual production prompts, group them by what they are really asking, and count how many of those groups your current cache is completely blind to. That number is usually bigger than people expect, and it is the clearest sign that an exact match cache has quietly stopped doing its job.