How Do I Use Memory in Long-Running Agent Loops?

Long-running agent loops — autonomous workflows where an AI agent operates for hours or days, making decisions, calling tools, and accumulating state — hit a wall that short interactions do not: the context window fills up. Once it does, one of three things happens: the agent drops early context and forgets decisions it made (context drift), the agent re-does work it already completed because it lost the memory of doing it (redundant work), or the agent errors out when the context window overflows. The fix is external memory: a persistent store the agent reads from and writes to throughout the loop, so accumulated knowledge survives context window eviction. Research from MemGPT (tiered memory management, arXiv:2310.08560), Voyager (lifelong learning via skill libraries, arXiv:2305.16291), and Reflexion (verbal reinforcement via episodic memory, arXiv:2303.11366) converges on the same architecture: agents need managed external memory — not bigger context windows — to operate coherently over long horizons.

The pain: what breaks when an agent runs too long

An agent starts a long-running task: refactor a codebase, research a market, process a dataset, or manage a multi-step deployment. It makes decisions, calls tools, reads results, and generates next steps. Each interaction adds tokens to the context window. After enough turns, the context window is full — and the agent's behavior degrades in three ways.

Context drift. As the context window fills, earlier context is evicted (or compressed, or simply lost in attention). The agent forgets the decision it made at step 3 — "we chose PostgreSQL" — and may contradict it at step 40. Packer et al. (arXiv:2310.08560) identified this as the core limitation of LLMs in extended interactions: "limited context windows, hindering their utility in tasks like extended conversations and document analysis." The solution they proposed — MemGPT's virtual context management — is essentially an external memory tier the agent manages itself, moving information between fast (in-context) and slow (external) memory as needed.

Redundant work. Without persistent memory, the agent cannot recall that it already tried an approach and failed. It re-attempts the same API call, re-reads the same file, re-runs the same analysis. Shinn et al. (arXiv:2303.11366) demonstrated this with Reflexion: agents that maintain an "episodic memory buffer" of past attempts — what worked, what failed, why — make better decisions on subsequent trials. The memory buffer is not a context window; it is a separate, persistent store of distilled lessons the agent reads before acting.

Decision amnesia. In a long loop, the agent makes a sequence of decisions: which approach to take, which file to edit, which API to call. Without memory, it cannot reconstruct why it made an earlier decision — it only sees the current context window. This leads to inconsistent behavior: the agent may undo its own earlier work, or take an approach that contradicts a constraint it established ten steps ago. Wang et al. (arXiv:2305.16291) showed that Voyager, a lifelong learning agent, solves this with an "ever-growing skill library" — a persistent store of executable skills the agent builds up over time. The skills are "temporally extended, interpretable, and compositional," which "compounds the agent's abilities rapidly and alleviates catastrophic forgetting." The skill library is external memory; the model's weights never change.

The architecture: memory as a separate module

The research literature on agent memory converges on a single architectural principle: memory should be a separate module from the model, not stuffed into the context window.

Zhang et al. (arXiv:2404.13501) organize the field around memory as "the key component to support agent-environment interactions" — the basis for "self-evolving capability." Their survey identifies multiple memory types (short-term, long-term, episodic, semantic) and multiple operations (read, write, reflect, forget), all operating on a store external to the model.

Sumers et al. (arXiv:2309.02427) proposed CoALA (Cognitive Architectures for Language Agents), a framework that describes agents with "modular memory components, a structured action space to interact with internal memory and external environments, and a generalized decision-making process." The memory is modular — it has its own structure, its own operations, and its own lifecycle — separate from the model's parameters or the context window.

Park et al. (arXiv:2304.03442) demonstrated this with generative agents that "store a complete record of the agent's experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior." The architecture has three layers: observation (what happened), reflection (what it means), and retrieval (what is relevant right now). Memory is not just storage — it is an active process of consolidation and recall that makes the agent more coherent over time.

Practical approaches to memory in long-running loops

1. Tiered memory (MemGPT pattern)

The agent manages memory in tiers: a small "core memory" in the context window (the current task, key facts), and a larger "archival memory" outside the context window (historical context, past decisions, learned lessons). The agent moves information between tiers as needed — promoting relevant archival memories to core when they become pertinent, and demoting stale core memories to archival when they are no longer active.

This is the approach Packer et al. (arXiv:2310.08560) demonstrated with MemGPT. The agent "intelligently manages different memory tiers in order to effectively provide extended context within the LLM's limited context window." The OS metaphor is apt: just as an operating system pages data between RAM and disk, the agent pages knowledge between context and external storage.

2. Episodic memory with reflection (Reflexion pattern)

The agent maintains an episodic memory buffer — a log of past attempts, outcomes, and verbal reflections. Before each new attempt, the agent reads the relevant reflections to avoid repeating mistakes and to build on past successes.

Shinn et al. (arXiv:2303.11366) showed that this "verbal reinforcement learning" — reinforcing the agent through linguistic feedback stored in memory, not through weight updates — produces significant improvements across coding, decision-making, and reasoning tasks. The key insight: the agent learns from its own experience without retraining.

3. Skill libraries (Voyager pattern)

The agent builds a library of reusable skills — executable code, procedures, or strategies — as it works. Each skill is stored externally and indexed for retrieval. When the agent encounters a similar task, it retrieves and composes existing skills rather than starting from scratch.

Wang et al. (arXiv:2305.16291) demonstrated that this approach enables "lifelong learning" — the agent "continuously explores the world, acquires diverse skills, and makes novel discoveries without human intervention." The skill library "compounds the agent's abilities rapidly and alleviates catastrophic forgetting" because skills are stored externally, not in weights.

4. External memory via MCP (the open-standard pattern)

The Model Context Protocol (MCP, specification 2025-11-25, modelcontextprotocol.io) is an open protocol — JSON-RPC 2.0 based, inspired by the Language Server Protocol — that standardizes how LLM applications connect to external data sources and tools. An MCP-compatible memory server gives any MCP-compatible agent runtime access to a persistent memory store.

The agent calls memory tools (recall, learn, forget, feedback) as part of its normal loop. Memory is not stuffed into the context window — it is retrieved on demand, like a database query. The memory store persists across sessions, across agent restarts, and across model switches. Open engram implementations like PLUR expose memory over MCP, so any agent runtime — Claude Code, Hermes, OpenClaw, Cursor — can read from and write to the same memory store.

What to store in long-running loop memory

Not everything belongs in the memory store. The signal-to-noise ratio matters — a memory full of irrelevant observations is as useless as no memory at all. The research suggests storing:

1. Decisions and rationale — what the agent decided, and why. This prevents contradiction and enables the agent to reconstruct its reasoning later.
2. Failed approaches — what was tried, what went wrong, what was learned. Reflexion (arXiv:2303.11366) showed this is the highest-value memory type for iterative improvement.
3. Skills and procedures — reusable code, validated workflows, successful strategies. Voyager (arXiv:2305.16291) showed this compounds agent capabilities over time.
4. Key facts — project context, user preferences, environment state that the agent needs to reference repeatedly but that does not fit in the context window.
5. Reflections — higher-level abstractions synthesized from observations. Park et al. (arXiv:2304.03442) showed that reflection — "synthesizing memories over time into higher-level reflections" — is what gives agents continuity and personality.

When to forget

Memory that grows without bound is its own problem. An agent that has accumulated 10,000 memory entries will retrieve noise alongside signal. The research points to several forgetting strategies:

• Temporal decay — memories that have not been accessed recently lose activation. This mirrors biological memory consolidation: frequently recalled memories are reinforced; unused ones fade.
• Feedback-driven relevance — memories that were useful (the agent acted on them and the outcome was positive) are promoted; memories that were not useful are demoted.
• Consolidation — multiple related observations are synthesized into a single higher-level reflection, and the raw observations are discarded. Park et al. (arXiv:2304.03442) demonstrated this: observations accumulate, then reflections are synthesized, then the agent retrieves reflections (not raw observations) for planning.

These forgetting strategies are not optional — they are what keep the memory store useful over long horizons. A memory system without forgetting is a log file, not a memory.

FAQ

How do I use memory in long-running agent loops? Use an external memory store that the agent reads from and writes to throughout the loop. The agent should store decisions and rationale, failed approaches, reusable skills, key facts, and synthesized reflections. Before each step, the agent retrieves relevant memories to avoid repeating mistakes, contradicting earlier decisions, or re-doing completed work. The memory store persists across context window evictions and across sessions. Research from MemGPT (tiered memory, arXiv:2310.08560), Reflexion (episodic memory, arXiv:2303.11366), and Voyager (skill libraries, arXiv:2305.16291) converges on managed external memory — not bigger context windows — as the solution.

What is context drift in AI agents? Context drift occurs when an agent's context window fills up and earlier context is evicted or compressed, causing the agent to forget decisions, constraints, or facts it established earlier in the task. This leads to inconsistent behavior, contradiction of earlier decisions, and redundant work. External memory prevents context drift by persisting key information outside the context window and retrieving it on demand.

How does MemGPT manage memory? MemGPT (arXiv:2310.08560) uses a tiered memory architecture inspired by operating systems: a small "core memory" in the context window for active information, and a larger "archival memory" outside the context window for historical context. The agent pages information between tiers — promoting relevant archival memories to core when needed, and demoting stale core memories to archival. This provides "virtual context management" that extends the effective context window.

Can AI agents learn from their mistakes during a long task? Yes — if they have memory. Reflexion (arXiv:2303.11366) demonstrated that agents which maintain an episodic memory buffer of past attempts (what was tried, what failed, why) and read from it before each new attempt produce significantly better results than agents without memory. The agent "verbally reflects on task feedback signals" and stores the reflection in memory — learning from experience without weight updates.

What is lifelong learning in AI agents? Lifelong learning is the ability of an agent to continuously acquire new skills and knowledge over time without forgetting what it already learned. Voyager (arXiv:2305.16291) demonstrated this with an "ever-growing skill library" — a persistent store of executable skills that "compounds the agent's abilities rapidly and alleviates catastrophic forgetting." The key: skills are stored externally, not in model weights, so new learning does not degrade existing capabilities.

How does MCP help with agent memory? The Model Context Protocol (MCP, modelcontextprotocol.io) is an open protocol that standardizes how agents connect to external tools and data sources — including memory servers. An MCP-compatible memory server gives any agent runtime access to persistent memory (recall, learn, forget, feedback) over a standard protocol. The memory store persists across sessions, agent restarts, and model switches. This means memory is not tied to one agent framework — it follows the agent across tools.