The Architecture of a Source-Backed Legal Intelligence System

Legal AI is having its GPT-3 moment. The technology is good enough to be useful, every founder is shipping, and the gap between the marketing demos and the production systems is starting to matter. This piece walks through the architecture of a source-backed legal intelligence system — what it actually takes to ship one that practitioners will trust.

The key constraint we imposed on the CourtGPT system: every claim the model produces must resolve back to a primary source the practitioner can verify in one click. That single constraint drives most of the architectural decisions below.

The four-layer architecture

A source-backed legal intelligence system has four distinct layers, each solving a different problem:

┌─────────────────────────────────────────────────────┐
│  Layer 4: Audit & Citation Interface                 │
│  (every claim links to its source, with date stamp) │
├─────────────────────────────────────────────────────┤
│  Layer 3: Grounded Generation                        │
│  (LLM constrained to retrieved sources)              │
├─────────────────────────────────────────────────────┤
│  Layer 2: Retrieval & Citation Graph                 │
│  (canonical citation form, crosswalks, AM tracking) │
├─────────────────────────────────────────────────────┤
│  Layer 1: Primary-Source Ingestion                   │
│  (statutes, regulations, case law, court rules)      │
└─────────────────────────────────────────────────────┘

The layers are loosely coupled but each one is independently useful — and each one is where most legal-AI products fail in different ways.

Layer 1: Primary-source ingestion

The first problem is straightforward: you can't cite what you don't have. The ingestion layer is responsible for normalizing primary sources into a canonical representation.

For CourtGPT, this means:

Federal level:

• United States Code (54 titles, ~60,000 sections)
• Code of Federal Regulations (~200,000 sections)
• Federal case law (Supreme Court, federal circuits, federal districts)
• Federal Rules of Civil Procedure, Criminal Procedure, Evidence, etc.

State level (all 50 states + DC):

• State constitutions
• State codes (statutes)
• State administrative codes
• State case law (state supreme courts, courts of appeal, trial-level where published)
• State rules of procedure

Local level (where published):

• Major municipal codes (San Diego, LA, SF, NYC, Chicago, Houston, others)

The total document count is roughly 1.8 million distinct primary-source documents as of mid-2026.

Ingestion pipeline challenges

The hard part isn't the volume — it's the variation. Each state publishes its statutes in different formats, different HTML structures, different naming conventions. Some publish PDFs only. Some have well-structured XML feeds. Some are accessible only via scraping.

We built a per-source adapter pattern: each primary source has its own ingestor that knows the source's quirks. The output is a normalized canonical form:

{
  "id": "cal-penal-code-187",
  "citation": "Cal. Penal Code § 187",
  "title": "Murder defined",
  "text": "...",
  "effective_date": "2024-01-01",
  "amendment_history": [
    {"date": "1995-01-01", "action": "amended", "summary": "..."},
    {"date": "2024-01-01", "action": "amended", "summary": "..."}
  ],
  "source_url": "https://leginfo.legislature.ca.gov/...",
  "jurisdiction": "California",
  "level": "state-statute"
}

The canonical ID is stable across re-ingestion. When a source is updated, we ingest the new version, generate a new amendment_history entry, and update the effective_date — but the canonical ID stays the same so existing references don't break.

Layer 2: Retrieval & citation graph

Once you have 1.8M documents, you need a way to find the right ones for a given query. This is where most retrieval-augmented systems spend their engineering effort.

We tried two approaches:

Approach A: Pure embedding-based retrieval. Embed every section, embed the query, cosine-similarity top-k. This works for "find me statutes about X" but fails for anything requiring precise citation matching. The problem: "compare choice-of-law statutes across three states" needs the system to identify the canonical choice-of-law statute for each state, which requires more than semantic similarity.

Approach B: Citation graph + embeddings hybrid. Build a citation graph keyed on canonical citation forms. Use embeddings for fuzzy matching and the citation graph for precise citation resolution. This is what we shipped.

The citation graph looks like:

[Cal. Penal Code § 187]  ──amends──>  [Cal. Penal Code § 187 (1995)]
       │
       ├──cited_by──>  [People v. Smith, 12 Cal. 4th 100]
       ├──cited_by──>  [People v. Johnson, 50 Cal. 3d 200]
       └──related_to──>  [Cal. Penal Code § 188]  (intent)

When the model needs to cite a statute, it queries the citation graph for the canonical form and gets a persistent identifier. When a statute is amended, the graph is updated atomically.

Why this matters for legal

Citation is the unit of legal reasoning. Statutes are cited by their canonical form, not by topic. Cases cite other cases by their canonical form. A system that can't reliably resolve "Cal. Penal Code § 187" to the canonical ID is a system that will produce wrong citations.

Layer 3: Grounded generation

This is the LLM layer, but with one critical constraint: the model can only produce claims that are supported by retrieved sources.

We didn't try to fine-tune hallucination out of the model. We constrained the generation space.

The implementation:

1. User query comes in.
2. Retrieval layer finds top-k relevant sources from Layer 2.
3. Sources are injected into the prompt as: "Use ONLY the following sources to answer. If the question cannot be answered from these sources, say so explicitly. Cite each source by its canonical form: [Citation 1], [Citation 2], etc."
4. Model generates an answer.
5. A post-processing layer verifies every [Citation N] reference in the answer maps to a source in the retrieved set. If any citation in the answer isn't in the retrieved set, the answer is rejected.
6. If the answer is accepted, the citation list is attached as a structured output alongside the prose.

The post-processing layer is essential. The model still hallucinates internally — but the output layer rejects ungrounded claims before they reach the user.

This isn't perfect. The model can still produce a sentence that sounds right but is subtly wrong. But the worst class of failure — a hallucinated citation — is structurally prevented.

Layer 4: Audit & citation interface

The final layer is what makes the system usable in practice. Every answer exposes:

• The answer itself
• The list of citations
• A click on any citation reveals the verbatim source text with the effective date
• A "version stamp" showing when the answer was generated

┌─────────────────────────────────────────────────────────────┐
│  Question: "What is California's murder statute?"           │
├─────────────────────────────────────────────────────────────┤
│  Answer:                                                     │
│  California defines murder as the unlawful killing of a     │
│  human being, or a fetus if the fetus has reached a         │
│  certain stage of development, with malice aforethought.    │
│  [1]                                                         │
├─────────────────────────────────────────────────────────────┤
│  Citations:                                                  │
│  [1] Cal. Penal Code § 187 — "Murder defined"              │
│      Effective: 2024-01-01                                  │
│      Source: leginfo.legislature.ca.gov                    │
└─────────────────────────────────────────────────────────────┘

When a senior partner clicks [1], they see the full text of Cal. Penal Code § 187 with its effective date. If they want to verify the cite, they can copy the citation into Westlaw or LexisNexis and confirm it matches.

This is the layer that builds trust. Without it, the system is just another chat interface.

What we got wrong

A few things we tried that didn't work:

Fine-tuning on legal corpora. We tried fine-tuning on a corpus of legal briefs. The model got better at legal-sounding language but worse at citation faithfulness. We abandoned this approach.

Pure LLM-as-judge for citation verification. We tried using a separate LLM to verify citations in another LLM's output. The verification LLM was about as reliable as the generation LLM — both hallucinated in similar patterns. We replaced this with the strict post-processing layer.

Vector databases with too-high dimensionality. We initially used 1536-dimensional embeddings (OpenAI ada-002) and the recall was fine for fuzzy queries but the precision for exact-citation queries was poor. We moved to a hybrid system with a lower-dim embedding for fuzzy and the citation graph for exact.

What this enables

Once you have this architecture, you can build a number of useful products:

• Citation-faithful legal research (the core CourtGPT product)
• Brief drafting with verifiable citations (output that lawyers can use as a starting point)
• Cross-jurisdictional comparison (the system's coverage enables side-by-side statute comparison)
• Privileged memo preparation (associates can use the system to draft memos with full audit trails)
• Compliance monitoring (the effective-date tracking enables alerts when a relevant statute changes)

The architecture is general. The constraint — citation faithfulness — is what makes it trustworthy for legal practice.

What's hard about this architecture

Three things:

Ingestion is ongoing. Statutes get amended, cases get published, codes get updated. The ingestion layer never finishes. We have a team that monitors sources for updates and re-ingests them on a regular cadence.

Citation graphs don't auto-build. The mapping from a statute in source-X format to the canonical citation form is non-trivial. Some sources are well-structured (state legislature XML feeds), others require significant NLP work to extract the canonical citation.

Audit interface is engineering, not design. It's not enough to display citations — they have to link to the verbatim source, with effective date, with version stamp, with the ability to copy into a brief. Every part of that interface is a feature.

A note on what we don't do

A few things CourtGPT intentionally doesn't do:

• Hallucinate authorities. The system cannot produce a citation that isn't in the source layer.
• Provide legal advice. The system is a research and drafting tool. The practitioner remains the lawyer of record.
• Speculate about how a court will rule. The system surfaces the controlling authority. Predicting outcomes is the lawyer's job.

This is a deliberate scope choice. The product is built to assist, not replace, the practitioner.

Closing

Building a legal-AI product is not primarily an ML problem. The ML piece is a relatively small part of the system. The hard parts are:

1. Building the primary-source ingestion pipeline
2. Building the citation graph
3. Constraining generation to be source-faithful
4. Building the audit interface that makes the system trustworthy

Get those four right and you have a useful product. Skip any of them and you have a chat interface that will eventually embarrass its users.

CourtGPT is built by Talking Machines LLC in San Diego, CA. Live product at app.courtgpt.ai. Technical inquiries: hello@courtgpt.ai.