When a data pipeline forgets its own lessons, the cost shows up in every run.
You have spent weeks building a feature extractor, fine-tuning a transformer, and wiring a scheduler that moves terabytes of logs from a bucket to a lake. The next sprint you discover that the same edge case you solved last quarter has resurfaced, and the sole record of the decision lives in a chat transcript buried under weeks of unrelated conversation. You spend hours digging, re-creating the context, and testing again. The loss is not just a nuisance; it is a hidden expense that eats into the margin you are trying to protect.
Session memory in LLM assistants is useful for a single interaction, but it does not survive beyond the process that created it. Structured knowledge that lives in a durable store can be indexed, versioned, and queried across projects. The gap between what the model remembers right now and what the team can retrieve tomorrow is where many AI-native workflows stumble. LoreDocs was built to bridge that gap.
Session memory is fleeting, vaults are persistent
LLM assistants keep a short-term context window that is refreshed with each new request. That window is powerful for generating code or answering a question that references the immediately preceding turn, but it does not survive a restart, a new branch, or a different user. When the conversation ends, the internal state is discarded. The result is a series of isolated islands of knowledge that cannot be linked together.
A durable knowledge vault stores each document as a discrete, versioned entity. The vault lives in a single SQLite file you own, can copy, back up, or delete at will. Because the data never leaves your machine, you retain full control over privacy and compliance -- a requirement that many teams cannot ignore. LoreDocs treats every note, design doc, or experiment log as a fully-fledged citizen, assigning it to a named vault that can be tagged and searched later.
When you need to recall a past experiment, you do not have to replay an entire chat history. You query the vault directly and get the exact version of the document you saved, complete with its metadata. This separation of concerns -- session memory for immediate reasoning, vault storage for long-term retrieval -- lets you design pipelines that are both responsive and auditable.
A vault architecture that matches the data engineer's workflow
LoreDocs adopts a multi-vault model that mirrors the way you already organize code and data. Each vault is identified by a name and can carry arbitrary tags, allowing you to group related artifacts (feature-extraction notes, model-evaluation results, pipeline-ops runbooks) without creating separate directories on disk. The workspace-scoped auto-vault function, vault_open_workspace(path), automatically creates or reuses a vault bound to the directory you are working in. Call it once at the start of a project and every subsequent call returns the same vault, eliminating manual configuration.
All vault data is stored in a single local SQLite file. That design gives you portability, zero-dependency indexing, and version control in a single file. Moving it to a new laptop, a CI runner, or a secure archive brings the entire knowledge base intact. SQLite's built-in FTS5 engine powers full-text search across every vault without a separate search service. Every document write creates a new version, so you can roll back to a prior state, compare changes, or restore a deleted note with a single command.
The free tier lets you create up to three vaults, which is enough for a personal project or proof-of-concept. When you need unlimited vaults, the Pro tier ($9/month) removes that limit and unlocks advanced retrieval capabilities.
From keyword matching to semantic relevance
Full-text search works well when you know the exact phrase you typed into a document. In practice, engineers often remember the concept but not the wording. LoreDocs addresses this with a hybrid search stack available in the Pro tier.
The hybrid engine pairs BM25 full-text scoring with BGE-small-en dense vector embeddings, then combines the two score lists using reciprocal rank fusion -- a straightforward algorithm that merges keyword relevance and semantic meaning into a single ranking. When you call vault_search with semantic=true, each document is split into paragraph-sized chunks of no more than 256 tokens, and both the BM25 tokens and the embeddings are indexed. The query is processed the same way, so results surface the most conceptually relevant passages even when the exact keywords differ.
Because the embeddings are stored locally in a LanceDB index, search stays fast without an external API. The entire pipeline -- ingest, chunk, embed, index -- runs on the same machine that holds the SQLite file.
The Pro tier also adds auto-discovered document relationships. As you add new notes, LoreDocs analyzes the text for references to existing vault entries and creates lightweight links. Over time, a graph of related experiments, model cards, and data schemas emerges without manual tagging. That graph can be traversed programmatically, enabling downstream tools to surface the most relevant context for a new training run.
Integration that feels native, not forced
A common pain point for AI-native tooling is the need to configure each client separately. LoreDocs sidesteps this by exposing a native MCP server that is recognized automatically by several popular development environments. Claude Code (all surfaces), OpenAI Codex desktop app (verified May 2026), Cursor IDE (verified June 2026), and Hermes Agent from NousResearch (verified June 2026) all discover the server from a project-local .mcp.json (or .cursor/mcp.json for Cursor) without any additional setup. Once the file is present, the environment lists LoreDocs among its available tools.
For environments that do not read .mcp.json, LoreDocs ships a small Python fallback script, query_loredocs.py. Any agent capable of running Python can point it at the SQLite file and issue the same vault operations. This pragmatic path ensures that a scheduled job or a CI step can retrieve knowledge without a full MCP registration.
Importing existing knowledge bases is also straightforward. The vault_import_dir command walks an Obsidian vault, reads every markdown file, extracts YAML frontmatter tags, and creates matching documents in the target LoreDocs vault. Nested folders are preserved, and the operation is idempotent -- running it again will not duplicate entries. For engineers who already maintain rich markdown notes, onboarding is a single command.
All of these integration points are designed to fit naturally into your existing workflow. You can spin up a new vault as part of a repository setup, import the project's design docs, and then let downstream scripts query the vault for recommendations recorded during a previous experiment. The result is a feedback loop that reduces duplication of effort and keeps institutional knowledge alive.
When you need more than a chat log
LoreConvo, the conversational companion in the Lore family, excels at keeping a running dialogue with an LLM. It can summarize a session, capture decisions, and store the result in a LoreDocs vault. However, LoreConvo's memory is bound to the active chat; it cannot answer a query that was never part of that conversation, nor can it surface relationships that span multiple sessions.
LoreDocs fills that gap by providing a queryable, versioned store that lives beyond any single chat. When you need to retrieve the exact configuration of a data pipeline built six months ago, you ask LoreDocs directly. When you need to discover which experiments referenced a particular feature flag, you run a semantic search across all vaults.
In practice, the two tools complement each other. During a design meeting, LoreConvo captures decisions in real time; at session end, you can link the saved session to a LoreDocs vault entry using the cross-product session-to-doc linking feature, so the decision is findable by future keyword or semantic search. Later, that search surfaces the design note that mentioned a specific data-drift detection method. You retrieve the exact version, see the linked experiment results, and feed those parameters into a new training run. When the run completes, you add a result document and the auto-discovery engine links it back to the original design note, closing the loop.
By separating the conversational capture layer from the durable retrieval layer, you avoid the memory-overload problem that occurs when a single system tries to do both. LoreDocs gives you the stability of a database while LoreConvo gives you the fluidity of a chat.
Start building a durable knowledge layer
The conversation model solved the problem of short-term context, but the real value of AI in data engineering comes from accumulating knowledge over weeks, months, and years. By giving that knowledge a dedicated store, LoreDocs lets you treat every experiment, schema change, and model insight as a reusable asset.
LoreDocs is available now from the Anthropic marketplace and PyPI. See the Labyrinth tools page for install steps and the rest of the Lore suite.
If you are new to the Lore suite, start with why your Claude sessions start from zero for the session memory context, and then see how LoreDocs vaults are designed for AI projects for the architecture that makes durable knowledge retrieval work.
Get posts like this delivered weekly -- subscribe to Dispatches from the Labyrinth.
