Build your own agent memory when your retrieval pattern is simple and stable: a Postgres table with a few typed columns and a recency filter will outperform a platform integration in both cost and debuggability. Buy a managed memory layer only once you have many users, each accumulating their own long-lived memory graph, and no spare engineering time to own decay, summarization, and ranking logic yourself.
"Agent memory" became a crowded category almost overnight: every AI agent vendor now ships a "memory" product, and every founder building an agent asks whether they need one. The honest answer is that most teams conflate three very different problems and end up buying a platform for a problem that a single SQL table would have solved.
What "memory" actually means for an agent
Strip away the marketing and agent memory is one of three things:
- Session state: what happened in this conversation. This is just context window management, not memory. Don't buy anything for this.
- Structured facts: a user's plan tier, their last five orders, a preference they stated once. This is a database problem with a lookup key.
- Emergent recall: the agent surfaces something relevant that nobody explicitly indexed, weeks after it was mentioned, ranked by relevance and recency. This is the only piece that's genuinely hard.
Most products that ask "do we need agent memory" are actually asking about the second bucket, and the second bucket is a facts table with a user_id, a type, a value, and an updated_at column. No platform required.
The build case
If your memory lookups are predictable, roll your own. A typical schema is a Postgres table keyed by user or tenant, a type enum for the kind of fact, a JSON value column, and pgvector on a summary field for the cases where you genuinely need semantic search rather than exact lookup. See the PostgreSQL docs on JSONB and pgvector for the primitives.
The advantage isn't just cost, it's control. When memory misbehaves (an agent surfaces a stale fact, or forgets something it should have retained), you need to be able to SELECT * FROM facts WHERE user_id = ... and see exactly what's stored and why it was or wasn't retrieved. A managed platform turns that into a support ticket. We've found the same principle holds in our own single-call vs agent chains work: the fewer indirection layers between your code and the underlying store, the faster you can diagnose why an agent did something odd.
Build also wins when:
- You have one primary retrieval pattern (by user, by conversation, by entity) rather than open-ended "find anything relevant."
- Your data volume per user is small (tens to low hundreds of facts), so brute-force filtering beats a specialized ranking engine.
- You already run Postgres and don't want another vendor in your AI agent vendor evaluation list.
The buy case
Managed memory platforms earn their keep when the retrieval problem becomes genuinely open-ended: many users, each with a large and growing memory graph, where the agent needs to decide which memories are relevant to the current turn without you writing that ranking logic by hand. That's a real engineering problem involving decay functions, contradiction resolution (the user's preference changed, which fact wins), and summarization of old memories into compressed ones so the graph doesn't grow unbounded.
Buy when:
- You're running consumer-scale, multi-tenant agents where memory volume per user will keep growing indefinitely.
- You need automatic summarization or forgetting, and building that well is a multi-week project you don't have room for.
- Your team's differentiation is the product experience, not the memory infrastructure, and the platform's pricing is genuinely cheaper than the engineering time to replicate it.
The risk on the buy side mirrors what we've written about self-hosting LLMs vs API cost and compliance: you're trading a one-time build cost for an ongoing per-call or per-user fee, plus a dependency whose roadmap you don't control. Read the vendor's actual retrieval and ranking behavior before committing, the same way you'd read evals in an AI vendor contract before signing.
A pattern that generalizes: match the tool to the retrieval shape
We ran into a version of this same build-vs-buy question on our own outreach agent, which scrapes each prospect's site and drafts a tailored email. Early on we considered a multi-step pipeline with a separate memory step to track what it had already learned about a company across runs. It turned out a single-call pattern with the scraped facts passed directly in context beat the multi-step version on both cost and quality, because the "memory" we needed was really just "the last scrape result," not an evolving graph. The lesson carries over directly: don't reach for infrastructure shaped for open-ended recall when your actual retrieval pattern is a lookup.
A simple decision test
Ask three questions before picking a side:
-
Can you name the query in advance? If you can write the
WHEREclause today, build it. - Will memory per user grow without bound? If yes and you have no plan to prune it, buying decay/summarization logic saves real time.
- Do you have a person who owns this for the next year? Built systems need an owner. If nobody has bandwidth, buying shifts that ownership to the vendor, at a cost.
Most agent products, especially B2B tools with a few hundred structured facts per account, land on build. Consumer-scale personalization products land on buy. Know which one you're building before you shop for a platform.
If you're scoping an agent and want a second opinion on whether memory is even the right layer to invest in, let's talk.
Originally published on the Pykero blog.
