A support dashboard started showing my QA tenant the production tenant's account summary, and my first thought was that the model had finally hallucinated customer data. But the wrong answer did not change between calls, which is not how a stochastic model behaves. It changed only after the cache expired, and that clue pointed away from the model entirely. I built this service around a free model endpoint because I wanted to keep infrastructure costs down without owning a GPU.
We used MonkeyCode's free model access and a free server option as the model backend for early traffic. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Symptoms that did not look like a cache bug
- Two tenants received identical replies even though their prompts differed at the end.
- The identical replies persisted for exactly 3600 seconds, then changed.
- Logs showed only one outbound model call during that window for many incoming requests.
- The response contained one tenant's account number and the other tenant's question.
What kind of bug is deterministic for an hour and then flips? That is a cache with a one-hour TTL, not a language model with a bad day. I had added Redis in front of the model to reduce latency and cut down on calls to the free endpoint, but I never treated the cache key as a contract. That was the whole bug.
The key was a lossy hash of the prompt prefix
Here is the function that caused the leak.
import hashlib
def cache_key_for(prompt: str) -> str:
# The first 80 characters of the prompt contain the shared system text.
return hashlib.sha256(prompt[:80].encode()).hexdigest()
Every request shared the same system prompt and opening template. The only differences appeared later, in the tenant ID and the customer account number. Truncating the prompt to the first 80 characters meant those differences never entered the hash. Two prompts from different tenants produced the same key, and Redis dutifully returned one tenant's generated answer to the other.
I reproduced the collision without touching a live model call.
tenant_a_prompt = (
"You are a support assistant. Read the ticket from customer account acct_7XQ "
"and summarize the last message. Customer account acct_7XQ"
)
tenant_b_prompt = (
"You are a support assistant. Read the ticket from customer account acct_ZP2 "
"and summarize the last message. Customer account acct_ZP2"
)
old_a = cache_key_for(tenant_a_prompt)
old_b = cache_key_for(tenant_b_prompt)
print(old_a == old_b) # True in the broken version
The broken key made two logically different requests look identical to the cache. Once I could reproduce that in a unit test, the fix became obvious instead of speculative.
Include every input that changes meaning, not just the full prompt
The corrected key names the dimensions that actually change the output.
import hashlib
def build_cache_key(
tenant_id: str,
system_prompt: str,
user_prompt: str,
model_version: str,
) -> str:
payload = "\x1f".join([tenant_id, system_prompt, user_prompt, model_version])
return hashlib.sha256(payload.encode()).hexdigest()
This key changes when any of the following changes:
- the tenant who submitted the request
- the system prompt or tool configuration
- the full user prompt, not a prefix
- the model version behind the endpoint
A cache key should be deterministic only within a request boundary. If two requests would legitimately produce different responses for the same user, they must not share a key. Prefix truncation broke that rule because it made the key lossy before any hashing could help.
When a shared cache is still the wrong tool
Even the corrected key does not make a shared Redis cache safe for every workload.
- A shared cache is dangerous when model responses contain names, account numbers, or other tenant-specific data.
- Personalized replies should be cached per tenant, per user, or in a tenant-isolated cache namespace.
- Non-deterministic generation settings should disable caching entirely.
- Auditing or compliance requirements may need a per-tenant store instead of a shared hash bucket.
Do not treat a better key as a substitute for tenant isolation. The key fixes accidental cross-tenant hits, but a shared cache still puts everyone's data in one place.
What I will test before the next release
I added a small regression test that builds keys for two tenants with identical prompts and asserts that the keys differ.
def test_cache_key_changes_when_tenant_changes():
base = build_cache_key("tenant_a", "sys", "summarize ticket", "v1")
other = build_cache_key("tenant_b", "sys", "summarize ticket", "v1")
assert base != other
The test would have caught the leak before any real data moved. The next step is to create a collision test with two prompts that share a long prefix but differ at the end, which is exactly how the production failure happened. If I add another model version or a new system prompt field later, the same test forces me to answer whether that dimension belongs in the key.
If your cache boundary does not have a test, production will eventually give you one in the form of a wrong answer. That is a much more expensive way to learn which input dimensions actually matter.
