I Benchmarked My Homelab Memory Stack: Hybrid Search + Local Reranker Took LoCoMo from 63% to 80%

Pure vector search got my agent memory stack to 63% on LoCoMo. Adding a sparse retriever and a reranker that runs on a card I already owned pushed it to 80%. The accuracy came from a stage that adds maybe 40ms per query, and the queries it fixed were exactly the ones I cared about: specific dates, error codes, and "who said what in which session" needles buried in months of conversation history.

If you're running a local agent that recalls facts across long conversations, this is the retrieval layer under everything else. A bad memory stack doesn't crash. It quietly hands the model the wrong three chunks and lets it confabulate a confident answer. That failure mode is worse than an outage because nothing tells you it happened.

The setup and why I benchmarked at all

My agents run on a memory stack I've written about before: a six-layer architecture for Claude Code with a wiki layer, a vector store, and an activation-based cognitive layer. The vector store is the workhorse. When an agent needs to recall a fact from a past session, it embeds the query, pulls the top-k nearest chunks, stuffs them into context, and answers.

That worked well enough that I never questioned it. Then I ran LoCoMo against it.

LoCoMo is a long-term conversational memory benchmark. It gives you multi-session dialogues that span hundreds of turns, then asks questions whose answers are scattered across those sessions. Single-hop lookups, multi-hop reasoning, temporal ordering, the works. It's a good proxy for what an agent memory system actually has to do, because the answer is never in the most recent turn. It's three sessions back, phrased differently than the question.

My vector-only stack scored 63%. Not terrible. Not good enough to trust an agent to act on. The interesting part wasn't the number, it was the shape of the failures.

What I tried first (and why it was the wrong lever)

My first instinct was the obvious one: the embeddings must be too weak. Swap the model, get better vectors, problem solved.

So I did the thing everyone does. I moved from a general-purpose embedding model to a larger, higher-ranked one on the MTEB leaderboard. Re-embedded the whole corpus. Re-ran LoCoMo.

63% went to 65%.

Two points. Hours of re-embedding for two points. That's when I actually looked at the failures instead of the aggregate score, and the pattern was obvious in hindsight. The questions I was getting wrong weren't semantically hard. They were lexically specific:

• "What was the ticket number the user mentioned?" — the chunk with TICKET-4471 in it wasn't in the top-k, because "ticket number" as a query embeds close to a hundred chunks that talk about tickets in general.
• "Which date did they say the migration finished?" — the model retrieved chunks about the migration, just not the one sentence with the actual date.
• "What did Maria say about the vendor?" — proper nouns get averaged into oblivion by dense embeddings. "Maria" and "the vendor" are needles, and cosine similarity is bad at needles.

This is the well-documented weakness of dense retrieval. Embeddings capture meaning, and they're great at "find me things about database migrations." They're bad at "find me the exact string TICKET-4471," because that string's meaning is thin. There's nothing semantic about an identifier. A better embedding model doesn't fix a problem that isn't about semantics.

The second thing I tried was cranking k. If the right chunk isn't in the top 5, pull the top 20. That helps recall, and it did nudge the score. It also blows up the context window with noise and triggers the "lost in the middle" problem, where the model ignores relevant chunks buried between irrelevant ones. I was trading a retrieval problem for an attention problem. Not a win.

The actual fix: sparse recall, then rerank for precision

The move that mattered was splitting retrieval into two jobs it was badly trying to do at once.

Recall is "get the right chunk into the candidate set somehow." Precision is "put the right chunk at the top." Dense search alone is mediocre at both for needle queries. So I stopped asking it to do both.

For recall, I added BM25 sparse search alongside the dense search and fused the two with Reciprocal Rank Fusion. BM25 is a keyword retriever from the 1990s, and it is still undefeated at finding exact tokens. TICKET-4471 scores high on BM25 the instant the query contains it. RRF combines the two ranked lists without needing to normalize their scores, which is the whole reason it's the default fusion method in every mature vector DB now.

Here's the hybrid retrieval, using LangChain's ensemble retriever over an Ollama-served embedding model and an in-memory BM25 index:

from langchain.retrievers import EnsembleRetriever, BM25Retriever
from langchain_community.vectorstores import Qdrant
from langchain_community.embeddings import OllamaEmbeddings

# Dense: semantic recall via local embeddings
embeddings = OllamaEmbeddings(model="bge-m3", base_url="http://10.0.0.100:11434")
dense = Qdrant.from_existing_collection(
    embedding=embeddings, collection_name="agent_memory",
    url="http://10.0.0.100:6333",
).as_retriever(search_kwargs={"k": 20})

# Sparse: exact-token recall for IDs, dates, proper nouns
sparse = BM25Retriever.from_documents(all_chunks)
sparse.k = 20

# RRF fusion. Weights lean slightly toward dense for this corpus.
hybrid = EnsembleRetriever(retrievers=[dense, sparse], weights=[0.6, 0.4])

That alone moved 63% to roughly 72%. The needle queries started landing in the candidate set. But they were landing at rank 11, or rank 8, not rank 1, and I was still pulling too many chunks into context to be safe. Recall was fixed. Precision wasn't.

For precision, I added a reranker. This is the part people skip because it "adds a model," and it's the part that did the heavy lifting.

A reranker is a cross-encoder. Instead of embedding the query and the document separately and comparing vectors (a bi-encoder, which is what your vector search does), it feeds the query and each candidate together through the model and scores their actual relevance. It's slower per pair, which is why you never use it for the first-stage search over thousands of chunks. But over 20 candidates? It's cheap, and it's dramatically more accurate because it can see the query and document at the same time.

I ran BAAI/bge-reranker-base locally. It's small, and it fits alongside my inference workloads on the Tesla P40 I already had without a fight over VRAM. Around 1.1GB loaded.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base", device="cuda", max_length=512)

def retrieve(query: str, top_n: int = 5):
    candidates = hybrid.invoke(query)            # 20-40 fused candidates
    pairs = [(query, doc.page_content) for doc in candidates]
    scores = reranker.predict(pairs)             # true relevance per pair
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in ranked[:top_n]]    # feed only the best 5

The pipeline is now: hybrid recall pulls 40 candidates, the reranker scores all 40, I keep the top 5. That top-5 goes to the model. LoCoMo landed at 80%.

Why this works, not just that it works

The reason the reranker earns its keep comes down to what a bi-encoder physically cannot do.

When you embed a document at index time, you compress its entire meaning into one fixed vector before you've ever seen the query. That vector has to be a decent answer to every possible question about that chunk. It's a lossy average. For a chunk that says "the migration finished on March 14th after Maria flagged the vendor delay," the embedding smears the date, the name, and the topic together. When your query is specifically about the date, the vector doesn't get any sharper, because it was frozen months ago.

A cross-encoder sees the query at scoring time. It reads "which date did the migration finish?" alongside that chunk and can attend directly to "March 14th." It's not comparing two averages. It's answering a specific relevance question with both halves in front of it. That's why reranking fixes precision on exactly the query types that dense search chokes on, and why a bigger embedding model didn't: the problem was never the quality of the average, it was the averaging itself.

Hybrid search and reranking are attacking two different failures, which is why stacking them compounds. BM25 guarantees the needle chunk exists in the candidate pool. The reranker guarantees it floats to the top of that pool. Neither one alone gets you there. BM25 without reranking dumps the needle at rank 9 with 19 distractors. Reranking without BM25 can only reorder a candidate set that never contained the needle to begin with. You need the recall stage to be generous and the precision stage to be strict.

This is the practical version of the theory I dug into in vector search vs activation-based recall: different retrieval mechanisms have different failure modes, and a serious memory system layers them instead of betting everything on one.

The latency tax, measured honestly

Nothing is free. Here's what the two-stage pipeline cost on my hardware, averaged over the LoCoMo query set:

Retrieval got roughly 4x slower in absolute terms and added about 55ms end to end. For an interactive agent where the LLM generation step is already 2 to 8 seconds, 55ms of extra retrieval latency is noise. Nobody perceives it. I paid 55ms and got 17 points of accuracy on the queries that decide whether the agent is trustworthy.

The trade would look different if I were serving retrieval as a standalone API at high QPS. Then 4x matters and I'd think about batching rerank calls or caching. For a single-user agentic workflow, it's the easiest 17 points I've ever bought.

One VRAM note, since the reranker shares a GPU with inference: bge-reranker-base at fp16 is small enough to coexist, but if your inference model already fills the card, you'll evict it or OOM. I keep the reranker pinned and size the LLM around it. On CPU it's viable too, around 200ms for 40 candidates on a modern core count, which is fine if you don't have spare VRAM.

Lessons learned

Look at the failures, not the score. The two hours I spent swapping embedding models were wasted because I optimized an aggregate instead of reading which questions I got wrong. The moment I bucketed failures by query type, the fix was obvious. Every point I gained after that came from a targeted change, not a bigger hammer.

Dense embeddings are bad at identifiers, and no embedding model fixes that. Ticket numbers, dates, SKUs, proper nouns, error codes. If your agent recalls anything with a specific token in it, you need a sparse retriever in the loop. This isn't a tuning problem. It's a property of how dense vectors compress meaning.

Reranking is the highest-use stage most people skip. It gets dismissed as "an extra model" and "more latency," and both are true and both are cheap. Splitting recall from precision is the core idea. Let the first stage be generous and dumb, let the second stage be strict and smart.

Build your own golden set. LoCoMo is a fine public benchmark, but the queries that matter for your agent are the ones your agent actually gets. I keep a small golden dataset of real recall queries and the chunk that should answer each one, and I run it on every change to the stack. Twenty good examples catch regressions that an aggregate score hides.

What surprised me most was how little the fancy part mattered relative to the boring part. I went in assuming the embedding model was the ceiling. The ceiling was a 30-year-old keyword algorithm and a small cross-encoder, both running on hardware I already had. This retrieval layer is the foundation the rest of the memory stack sits on, and it's the same layer I'd want solid before wiring agents together into anything multi-agent. Since the reranker runs locally, none of the recall traffic leaves the box, which keeps the whole thing aligned with a privacy-routed inference setup instead of shipping every query to a hosted reranking API.

If you're building agent memory or predictive systems on your own hardware and want a second set of eyes on the retrieval layer, that's the kind of work I do at GuatuLabs. The stack is simpler than the marketing around RAG makes it sound. Two retrievers, one reranker, and the discipline to measure what you actually broke.