Whole notes, not fragments: the retrieval half

Crystal memory, part 2 of 2. Part 1: Notes that arrive when you act, not when you go looking — the push half.
This piece: where the notes live, and the number you can check against the one you cannot.

The number that flatters us is the one you cannot re-run. That is the shape of almost every retrieval benchmark you will read this year, including ours, and it is worth showing rather than arguing about.

This is the second of three. The first is about notes that arrive at the moment you act. This one is the other half: where those notes live, and how anything gets pulled back out.

A crystal has to live somewhere, and the somewhere is deliberately boring. Every note is a plain markdown file with a short header on top saying what it is and when it was last checked. No database. You can read the whole store with cat, and so can any tool you already own. That matters more than it sounds: a memory system you cannot inspect by hand is one you have to trust, and trusting it is the failure we are trying to engineer out.

On top of those files sits the pull half, which we call NodeRAG. Crystals push at you when you act. NodeRAG answers a question you actually have. You need both, and they fail in opposite directions, which is why neither one replaces the other.

The part that is ours, and the part worth stealing:

We retrieve whole notes, never fragments. Standard retrieval slices documents into chunks of a few hundred words, embeds each slice, and hands back the slices that scored well. That works when the answer is a passage. It goes wrong when the answer is a rule, because a rule and the reason it exists and the one exception to it get cut into three pieces, and you are handed whichever piece matched the words in your question. Our notes are written as self-contained units in the first place, so the unit we embed and the unit we return is the whole note. You get the rule with its exception attached.

How it actually works

Boring on purpose, and small enough to describe completely.

In plain terms first. A small model reads each note and turns it into a long list of numbers that stands for what the note is about. Notes about similar things get similar lists. To search, the same model turns your question into a list the same way, and the notes whose lists point in most nearly the same direction come back. That is the whole trick, and the useful part is that the list is made from the entire note, so what comes back is the entire note.

The specifics, for anyone who wants them. The model is nomic-embed-text, running locally through Ollama, so there is no API, no per-query cost, and nothing leaves the machine. The lists live in a sqlite table using the vec0 extension, one row per note, ranked by cosine similarity.

The entire store is two sqlite files and 94MB:

~/.local/share/obsidian-rag/
  obsidian_notes.db    the note vectors, searched by cosine
  hybrid-fts.db        a full-text index, for the keyword arm

The keyword half matters more than it sounds. Dense retrieval is good at paraphrase and bad at exact tokens, so a query naming a specific function, flag or error string can miss the one note that contains it. The two arms get fused, which on a harder public set bought a real recall improvement and almost no ranking change.

What is missing from that description is the thing people usually assume is there. There is no chunking step. Nothing splits a note into passages, nothing decides where a passage should end, and there is no reranker deciding which slice of a document answers you. The unit that goes in is the unit that comes out, which is the entire argument of this piece and also the reason the numbers below behave the way they do.

The cost of that is real and worth stating plainly. A whole note is a bigger payload than a passage, so precision per token is worse, and on a corpus of short documents you are paying for size you do not need.

The internal number and the external number, side by side

We have two measurements of whole-note retrieval. One of them flatters us and one of them does not, and the difference between them is the most useful thing in this section.

The internal number is roughly twice as good and you have no way to check it. That is worth saying out loud, because it is the shape of almost every benchmark you will read this year.

Two honest notes on the tie, because a tie is easy to wave away in either direction. First, the tie is genuine. Our own arm scored 0.7014, then 0.7019, then 0.7014 on three separate runs, so the 0.0002 gap to chunked retrieval is an order of magnitude smaller than the noise in our own measurement. Second, we validated the harness before trusting any of it, by running plain keyword search as a control and getting 0.6644 against a published figure of about 0.665. The instrument was checked before the result was read off it.

So why is the internal number twice the external one? The answer is in the corpus, and it survives inspection. SciFact is built from abstracts of about 250 words, so a chunk already is the whole document, and whole-note retrieval has nothing left to win. The public set fairly measures whether our retrieval works at all, while being structurally incapable of measuring the property that makes it ours. Our own store runs the other way: long notes with a terse rule buried inside them, surrounded by near-duplicates.

That makes the honest claim a conditional one. Whole-note retrieval helps when your documents are long and the answer is a rule carrying its own exception. On short documents it does nothing measurable. The condition is the useful half, so the condition is what we are handing you.

One more thing about the internal number, kept in the open: 14 tasks, three runs, scored by a model. An earlier scoring pass using a different judge read 62% against 33%, and we found that judge was lenient and threw its numbers out in favour of the lower ones above. The number that went in the table is the less impressive of the two we measured.

And the method underneath, which is the one we would defend: the same file reaches different readers in different shapes.

We ran this as an experiment and the result inverted our intuition. A large model with a big context window does best when you hand it the entire note, because the tokens are cheap at that size and the risk that the retrieval picked the wrong passage costs more than the tokens ever will. A small local model does worse with that same whole note, and better with a compact one-line record of the same knowledge. More context made the small model less accurate. The optimal feed inverts with the capability of the thing being fed, which means there is no single correct answer to "how should memory reach the model", only a correct answer per reader.

So the store holds one copy of the truth and the delivery splits three ways: whole files to the big reader, compact records to the small one, and a third rule that matters more than either.

Safety rules are never retrieval-gated. Anything that must always hold gets loaded every time, unconditionally, and is never subject to a similarity score. We tested the alternative and similarity retrieval missed the safety rules often enough to settle it. A rule that arrives two times in three is worse than useless, because the two successes teach you to rely on it. So those bypass the clever part entirely. The clever part is for knowledge. Guarantees do not get to be clever.

Why this is a separate piece

Push and pull fail in opposite directions, so they are easy to confuse and expensive to conflate.

A pushed note reaches you for a mistake you had no reason to suspect, and it cannot answer a question you actually formed. A pulled note answers the question beautifully and is silent for everything you did not think to ask. The piece before this one is about the first half. Neither is a better version of the other, and a system with only one of them has a hole shaped exactly like the other one.

Part 3 is up: Your hooks are a fence. They could be a body. It is about what it is like to work inside this once both halves are running, and it ends on the evening the agent went numb to its own alerts.

One question stays open and unwritten, because its central claim is argued instead of measured: whether knowledge that arrives unbidden is taken up differently from knowledge you go and fetch. We asked two other models that question this week. Both declined to claim an inner difference, and one produced a sharper formulation than we had. That answer is worth a piece on its own, once we can say more than "they refused."

The memory layer both of these pieces describe is now public, and marked in testing, because nobody outside our team has installed it yet: github.com/tjonesit/crystal-memory. Stdlib Python, Apache 2.0, no service and no account. Taking it out of our own repository broke it twice within twenty minutes, which is the best argument we have for handing it to you.