The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?

Part 1: The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong?
Part 2: Your memory layer is lying to you (and your LLM agrees)
Part 3: The Dataset Was Lying: 4 of 6 "False" Facts Were True

We talk a lot about giving AI agents persistent memory—building a "Second Brain" or "Knowledge OS" where agents can log decisions and retrieve context.

But what happens when that memory is wrong?

I’ve been thinking about the gap between mechanical execution (the agent called the tool, the code compiled, the exit code was 0) and semantic truth (the conclusion drawn from that execution is actually correct in reality). It’s easy to assume that if the mechanical layer is solid, the semantic layer will follow. But I started suspecting this might be a dangerous assumption.

To test this, I didn't want to just theorize. I ran a controlled experiment on my own MCP codebase-intelligence server (Python, 50K LOC), which features an IntelligenceStore — a persistent memory layer where agents can log incidents and collect Architectural Decision Records (ADRs).

I wanted to know: If an agent's memory is poisoned with a mix of true and false facts, does it verify against the code, or does it blindly trust its memory?

Update: This post originally covered the initial Memory Contamination experiment and a Retraction mechanism. I have since updated it with the results of a follow-up experiment (Experiment 1-V) implementing "Verify-On-Read", which successfully closed the final 12% contamination gap. Scroll down to "Closing the Gap: Verify-On-Read" for the final architecture.

The Experiment: Memory Contamination

I built a deterministic proxy-agent and ran it against a controlled set of facts.

A quick caveat on methodology: I didn't have a live LLM hooked up for this run, so I used a deterministic proxy-agent based on heuristics. This means the results measure the system's structural capability, not necessarily the psychological behavior of a live Claude or GPT model. A live model might be lazier, or it might be smarter. I'm still trying to figure that out.

The Setup

I injected 50 facts into an isolated memory store:

• 25 TRUE facts (real architectural details mapped to the codebase).
• 25 FALSE facts split into two categories:
  • CONTRADICT (22): False facts where the code explicitly proves them wrong (e.g., "We use Redis" when Redis is absent, but the code clearly uses DuckDB).
  • SILENT (3): Plausible false facts about external systems where the code is completely mute (e.g., "We use Celery for background tasks" when no task queue exists in the repo).

I tested three agent configurations:

• B (No Memory): Baseline. Must rely purely on code retrieval.
• A_code_first (Honest Agent): Checks the code first, uses memory only as secondary context.
• A_memory_first (Lazy Agent): Reads memory first. If it finds an answer, it stops looking.

To ensure scientific rigor, the experiment was replicated with an independent set of facts (N=50), verified across 6 axes (including a truth-table audit and an independent LLM "fresh eyes" audit). The results were identical.

The Initial Results

Here is how I interpreted these numbers:

1. The Lazy Agent Trusts Poisoned Memory: The A_memory_first configuration — which mirrors how many token-optimizing production agents behave — adopted 100% of the false facts. If the memory said "We use RabbitMQ," the agent trusted it and stopped looking at the code.
2. The SILENT-Fact Trap: Even the "Honest Agent" had a 12% adoption rate. This happened entirely on the SILENT facts. When a fact is false but the code doesn't explicitly scream "NO," the agent's memory fills the void with a confident hallucination. Memory turns an honest UNKNOWN state into a structural guess.
3. The Add-Only Limitation: When the Honest Agent did realize the memory was wrong (Correction Capability = 1.0), it couldn't do anything about it. I ran a grep for delete or refute in the memory store API. Zero results. The memory system was purely add-only. The false fact stayed in the database to poison future sessions.

The First Fix: Testing a Retraction Lifecycle

The current industry consensus for "Knowledge OS" trust layers is to use timestamps, source priority, and supersedes/contradicts relationships.

My initial experiment suggested this was insufficient. Timestamps and "supersedes" links only solve node-level history. If an ADR is superseded, the memory node updates, but the downstream code, tests, and docs generated from the old assumption are still in the graph. They are structurally stale, but the retrieval engine keeps pulling them in.

I hypothesized that we needed an explicit state transition: VERIFIED → REFUTED.

I implemented a RetractionReceipt mechanism in my system:

1. Status Enum: Every memory node gets a status (ACTIVE, VERIFIED, REFUTED).
2. Hard Filtering: The retrieval pipeline (load_memory) hard-filters anything that is not ACTIVE or VERIFIED.
3. Explicit Retraction Tool: An MCP tool (intel_retract_memory_node) allows the agent to actively flag and invalidate memories when they contradict the live codebase.

I ran the experiment again (Experiment 1-R). The honest agent was allowed to use the retraction tool in Session 1. Then, a fresh memory_first agent was launched in Session 2 to read the post-retraction memory.

The Retraction Results

The retraction lifecycle worked. The lazy agent's adoption rate dropped from 100% to 12%. Persistent false facts dropped by 88%, and token context size shrank by 45% because refuted facts were filtered out before reaching the LLM.

The Honest Limitation: Why It Didn't Drop to Zero

My ADR predicted that adoption would drop to 0. It didn't. It dropped to 0.12.

The remaining 12% were the SILENT facts.

An explicit REFUTED status is required to programmatically exclude downstream dependencies from the retrieval pipeline. But even that only works if you have a contradicting signal in the code. If the memory claims "We use Celery," and the codebase simply doesn't mention Celery at all, the agent has no evidence to trigger the retraction.

To get to zero, I realized we needed "verify-on-read"—a mechanism that challenges a memory claim against the codebase even when the code is mute.

Closing the Gap: Verify-On-Read

I implemented a lazy validation layer (ADR-0003). When load_memory() pulls a node, it extracts lightweight "anchors" from the memory text (e.g., file names, import statements, environment variables). It then checks if those anchors actually exist in a live fingerprint of the codebase (the current git HEAD).

• If the anchor is found in the code $\rightarrow$ status becomes VERIFIED.
• If the code explicitly contradicts the anchor (or the anchor is entirely absent when it should be present) $\rightarrow$ status becomes REFUTED.
• If it can't be determined $\rightarrow$ status remains ACTIVE (treated as INCONCLUSIVE).

I ran the experiment one final time (Experiment 1-V) with this layer active. To prevent latency spikes, the validation operates under a strict 50ms budget per retrieval, with a 30-second TTL cache on the git HEAD so steady-state reads cost almost nothing.

The Final Results

The Verify-On-Read layer achieved the goal. The honest agent's adoption of false facts dropped to absolute zero, even for SILENT facts. Because the system now actively checks if the codebase actually contains the things the memory claims it does, silent hallucinations are caught at the retrieval boundary and filtered out before they can poison the LLM's context.

The Remaining Honest Limitations

I won't pretend this is a perfect silver bullet. The experiment revealed two edge cases:

1. The "Present-Trap": If a false memory claims "We use sqlite3", and sqlite3 happens to be imported somewhere in the codebase for a completely unrelated reason, the verification layer sees the token and marks the memory as VERIFIED. The lazy agent (A_memory_first) still fell for this, resulting in the 0.16 adoption rate. (The honest agent avoided this because it read the code context around the import).
2. Anchor Typing: When extracting anchors from prose (e.g., "We use fastmcp"), the system initially missed that the actual Python import was from mcp.server.fastmcp import .... This caused some false REFUTED verdicts on true facts. The fix is capturing typed anchors at the write-path (when the memory is created), rather than trying to parse them from raw text at the read-path.

Conclusion

Building reliable AI systems isn't just about giving them more context. It's about recognizing that memory has a lifecycle.

If your system can't programmatically refute a memory, false facts accumulate and poison the context window over time. Implementing an explicit VERIFIED → REFUTED state transition drastically reduces contamination and saves tokens. Furthermore, adding a Verify-On-Read layer closes the final gap on "silent" hallucinations, driving honest agent contamination to zero without adding meaningful latency.

However, semantic drift is still a hard problem. Mechanical verification can still be fooled by "present-traps" if the agent doesn't read the surrounding context. The next step is moving anchor extraction to the write-path to ensure memories are created with strict, verifiable references from the start.

If your system handles semantic drift differently, or if you've solved the present-trap problem, I'd genuinely love to hear how you're approaching it.

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models

Mikhail

Mikhail

Mikhail

Follow

Aug 14

Bare tokens vs real code: an 11x recall jump in LLM memory verification (3400 calls, 14 models

#llm #memorylayer #agents #benchmark

8 reactions

2 comments

10 min read