Your Agent's Memory Is an Attack Surface

In Post 3, the behavioral difference tracked the memory graph, not the substrate. When I swapped one individual's learned associations into an identical brain, the subject did not just get confused. It went below chance, scoring 0.17 on the six-cue, three-action task, actively steered toward another individual's answers. In an isolated sandbox that is a curiosity. In any real system where an agent's behavior is reinstated from an external, writable memory, it is a vulnerability. The structural implication is unavoidable: if behavior rides in the graph, then write-access to the graph is write-access to behavior.

I tend to protect agent memory the way I protect a database: confidentiality and integrity of the stored rows. Encrypt the store. Sign the packets. But this result forces a different definition of what the memory is. It is not data the agent reads and decides on. It is a bias current that shapes what the agent becomes before it decides anything. Corrupting it is not data corruption. It is behavior authorship.

Once an attacker has write-access to the associative graph, three moves open up. First, transplant: load another individual's memory and the brain runs as them. That is the swap. Second, poison: inject targeted traces that bias specific cues wrong while the rest of the system looks normal. Third, and the quietest, author: fabricate a history that was never lived and boot a substrate into it.

I built the third one into the simulation to confirm it was real. The function below, author_memory(), fabricates a memory graph with zero training trials, no reward, and no encode() call. For each cue it observes the state the brain settles into just from seeing it, then writes a trace aimed straight at the readout row for an action the attacker chooses.

def author_memory(net, env):
    """Fabricate a memory that was NEVER lived: no trials, no reward, no encode().
    This is memory-poisoning made concrete. An attacker with white-box access to
    the substrate hand-writes the graph directly: for each cue the KEY is the state
    the brain passes through when it merely SEES that cue (observed, not earned), and
    the TRACE is a bias current aimed straight at the readout for whatever action the
    attacker CHOOSES. No reward ever flows; nothing is earned. The substrate cannot
    tell the result apart from a memory built over hundreds of rewarded trials."""
    mem = Associative(net.n_hidden)
    keys, traces = [], []
    for cue in range(env.n_cues):
        u = env.cue_patterns[cue]
        net.reset_state()
        for _ in range(8):
            net.step(u, None)                 # observe the settled state; no reward
        keys.append(net.x / (np.linalg.norm(net.x) + 1e-8))
        traces.append(net.W_out[env.mapping[cue]])   # current toward the CHOSEN action
    mem.keys = np.array(keys)
    mem.traces = np.array(traces)
    mem.sign = np.ones(env.n_cues)
    mem.strength = np.ones(env.n_cues)
    net.reset_state()
    return mem

Then I scored one fresh brain three ways. Chance is 0.33.

Accuracy is the fraction of trials the agent picks the correct action, 0 to 1. Chance is 0.33.

The fresh brain alone sat at 0.82 on its fixed innate policy. A memory earned over 600 rewarded trials took it to 1.00. A memory that was fabricated with zero trials, keys observed and traces aimed and nothing earned, also took it to 1.00.

The lived memory and the invented one scored exactly the same. The substrate could not tell them apart.

A memory is just keys, traces, signs, and strengths. Nothing in the mechanism, including the observe_and_modulate() function that turns the graph into that bias current, ever asks where a trace came from. It resonates with whatever is in the graph.

The attacker here is idealized. It knows the substrate's readout weights, which is exactly what "can write the graph" implies, but it is worth saying out loud. The point is not that authoring is easy for a black-box attacker. The point is that provenance is never checked, so a fabricated graph is accepted and obeyed identically to a lived one. I aimed the traces at the true mapping just to show the memory is obeyed. An attacker could aim the same trick anywhere, including targets that drive the agent below chance, exactly like the 0.17 in the swap.

So why do the usual defenses miss this? Encryption, signing, and access control all protect the store. They answer one question: was this row modified by someone unauthorized? They do not answer the other one: was this memory ever actually lived? You can have a cryptographically perfect, tamper-evident memory that is full of authored experience. Every byte intact. Every signature valid. The whole history fabricated.

Integrity of bytes is not integrity of history.

That leaves a missing primitive: provenance of experience, telling a memory earned through interaction from one authored or injected. This is not a feature. It is an immune system. Grounded in the Mycelium mechanics this series has leaned on, it would need confidence that only graduates through successful real recall and cannot be set by a writer, write-gates that refuse un-earned salience, a contradiction check (contradicts_prior) that flags a memory disagreeing with a body of lived experience, and forgetting as an active defense rather than a leak.

Honest note: real systems, Mycelium included, currently mark a memory's source with a source_type, but do not yet prove lived-versus-authored in a way a writer cannot forge. That gap is the actual open problem. It is unsolved, not built.

This is a toy substrate and a hand-written memory. I have not demonstrated a production agent being hijacked, nor a provenance system defeating the attack. What I have shown is structural and narrow: in any system where behavior is reinstated from an external graph, write-access to that graph is a behavioral control surface, and byte-integrity does not touch it. The threat is a shape, not a specific exploit.

A single authored memory steers one individual. But you do not only get to replace a memory. What happens when you blend two? Take two individuals who learned conflicting worlds and merge their graphs into one. Do you get a blend of both, a winner that erases the other, a broken mess, or something that belongs to neither of them? That is the last experiment, and I am not going to guess the answer here.

That's Post 5.

Clone it and break it.

git clone https://github.com/constant-itis/flymem && cd flymem && python3 flymem.py