LLM Knowledge Base GitHub Template [2026]: Starter Kit Repo

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

You can have a working llm knowledge base GitHub template in about 45 minutes. Not a slide deck. Not an architecture diagram you’ll never build. A real, forkable repo with a folder structure, ingestion scripts, CI checks, a tiny eval harness, and an agent-readable layer (llms.txt, prompt packs, conventions) so the thing stays useful after sprint 1.

Here’s the thing nobody wants to admit: most teams don’t have a “knowledge base.” They have a graveyard. A RAG prototype ships in week 1. By week 4 the “KB” is a junk drawer. Broken links. Duplicated docs. Mystery PDFs. And zero signal on whether retrieval got better or quietly face-planted.

This post is my opinionated starter kit. Copy the structure as-is. Swap the internals (ingestion + indexer) for your stack. Keep the discipline.

Here’s the official video that pushed this direction for me. Obsidian + Claude Code workflows are becoming the default in a lot of teams:

[YOUTUBE:KK4e1puhaEw|How To Build an LLM Knowledge Base in Obsidian with Claude Code]

What is an LLM knowledge base?

An LLM knowledge base is a version-controlled collection of source material plus automation that turns it into retrieval-friendly artifacts (normalized text, metadata, indexes) so a large language model can answer questions with citations reliably.

A wiki or docs site is optimized for humans browsing. An LLM knowledge base is optimized for machines retrieving. That sounds subtle until you’re dealing with the annoying edge cases: stable chunk boundaries, provenance, license metadata, redaction, and tests that fail your PR when you break grounding.

When I’m building production AI features, I treat the knowledge base like a subsystem, not “content”:

• Inputs are explicit (where did this text come from?)
• Transforms are repeatable (ingestion is code)
• Outputs are testable (retrieval quality has regression tests)
• Freshness is automated (scheduled jobs, not “someone remembers”)

If you’re doing retrieval-augmented generation, your “KB” is not a folder of PDFs. It’s a pipeline.

The starter kit repo structure (and why it’s shaped this way)

If you want this to survive a real team, stop mixing raw sources, normalized docs, and indexes in the same folder. That’s how you end up with “which file is the truth?” debates, and then everyone stops trusting the system.

This is the layout I recommend. It’s intentionally boring:

Two strong opinions:

1. content/ is the single source of truth. Your site builds from it. Your RAG index builds from it. No duplication.
2. index/ never gets committed. It’s an output artifact. Commit it and you’ll fight merge conflicts and repo bloat forever.

Running this blog’s multi-agent publishing pipeline taught me that deterministic gates beat “we’ll notice later.” I’ve got 261+ published posts flowing through automation, and the stuff that breaks is always the boring plumbing: link rot, formatting drift, inconsistent templates. So I design KB repos the way I design build systems. Fail early. Make the failure loud.

Minimum viable files you should ship on day 1

These are the files I’d include in the first commit of an llm knowledge base template repo:

1. README.md with a 10-minute quickstart
2. content/ with 2–3 example pages
3. metadata/provenance.yml schema + one filled example
4. pipelines/ingest.py (or ingest.ts) that produces Markdown + metadata
5. .github/workflows/ci.yml with lint + link check + eval smoke tests
6. .github/workflows/freshness.yml scheduled weekly re-ingestion
7. agent/llms.txt + agent/system-prompt.md

That list is your starter kit contract. Everything else is negotiable.

Ingestion: turn markdown/html/pdf into normalized markdown

The entire point of this repo is to turn messy inputs into predictable Markdown.

My ingestion rule is simple: if it can’t become clean Markdown, it doesn’t belong in content/. Keep the raw thing in sources/ if you need it for traceability. But your canonical layer needs to be diffable, reviewable, and boring.

What file types are best (Markdown vs PDF)?

Markdown wins for three reasons:

• It’s diff-friendly in PRs.
• It’s easy to chunk consistently.
• It’s easy to attach metadata inline (frontmatter) without inventing a new database.

PDFs are fine as sources, but treat them like compiled artifacts. Extract and normalize into Markdown, or you’ll end up embedding garbage text with broken ordering, orphaned headers, and “Page 12 of 40” sprinkled everywhere.

A practical trick: OpenAI’s docs explicitly say that Markdown versions of doc pages are available by appending .md to the URL and they point to an llms.txt index (OpenAI docs). That’s exactly the shape you want for automated ingestion. If a vendor doesn’t offer something like this, your ingestion cost goes up. A lot.

A simple ingestion pipeline you can actually maintain

I’m not going to sell you a single “one true” stack. I’ve watched teams drown in fancy pipelines they can’t debug. The boring flow below keeps working:

1. Fetch from canonical sources (official docs, internal Markdown, ticket exports)
2. Normalize into Markdown with consistent headings and frontmatter
3. Scrub secrets/PII before anything touches embeddings
4. Chunk deterministically (same input => same chunk IDs)
5. Index (embeddings + vector DB) from content/ only

Frontmatter fields I like, because they force provenance instead of vibes:

• source_url
• retrieved_at (ISO timestamp)
• license (SPDX identifier if you can)
• owner (team/person)
• pii (none|redacted|contains_sensitive)

Concrete number: set a default chunk size like 800–1,200 tokens (or ~3–6 Markdown paragraphs) and keep it stable. Changing chunking is a breaking change. Treat it like an API.

GitHub Actions automation to prevent knowledge-base rot

A knowledge base that needs a human to remember to run scripts is already dead.

GitHub Actions workflows are made for this. GitHub’s own docs define workflows as automated jobs triggered by events like pushes and pull requests (GitHub Docs). Use them as guardrails, not decoration.

I’d ship four gates from day 1:

• Markdown lint on every PR
• Link check on every PR (internal + external)
• Ingestion smoke test on every PR (can we regenerate content/?)
• Scheduled freshness run weekly (re-fetch sources, open PR if diffs)

Concrete number: run scheduled ingestion weekly for external docs, and daily for fast-moving internal sources (runbooks, incident playbooks). If you do it “monthly,” you’ll spend the first week of every month relearning your own system.

This is also where teams get sloppy with secrets. Don’t. If you need a reference setup, my gitleaks + pre-commit + CI setup is the exact style of guardrail you want around ingestion scripts.

“Validate docs changes” PR checks that matter

If I had to pick only two checks for a github template repository for documentation, I’d pick:

1. Dead link detection (because link rot is guaranteed)
2. Provenance enforcement (because mystery docs destroy trust)

A policy that works in practice: any file in content/ must have source_url and retrieved_at. If it doesn’t, CI fails. No exceptions. If someone wants to paste “tribal knowledge,” they can put it in a draft area. The canonical KB needs receipts.

Make it agent-readable: llms.txt, prompt packs, and conventions

2026 reality: your “knowledge base” isn’t just for humans. It’s for AI agents and coding assistants.

That means you need an explicit interface. Not a bunch of implied folder magic that only the person who set it up understands.

What is llms.txt and how do I add it to my docs?

llms.txt is a convention for publishing a machine-readable index of your documentation and important entrypoints. OpenAI’s docs call it out directly as the “complete documentation index” pattern (OpenAI docs).

In the starter kit, I put it in agent/llms.txt and publish it at the site root (e.g. https://yourdomain.com/llms.txt).

Also ship:

• agent/system-prompt.md: what the assistant is allowed to do and how to cite
• agent/tool-specs/: tool descriptions for function calling (keep them versioned)
• agent/conventions.md: how to add a page, how to name files, how to mark deprecations

Concrete number: keep your “how to add a page” recipe under 15 lines. If it’s longer, people will freestyle. Freestyling is how you get six naming conventions and three “final_v2” folders.

If you want more templates, I’ve already written up my agent readable documentation toolchain and a set of AI-Readable Documentation templates. Same goal. Assistants that can navigate your repo without inventing structure.

Prompt engineering is not where you start

A lot of teams start by prompt-tuning the assistant when the KB is a mess. That’s backwards.

Anthropic’s docs are blunt about the right order. Have clear success criteria and evaluations before you iterate on prompts (Anthropic prompt engineering overview). That’s why this starter kit treats evals as a first-class folder, not a “phase 2” task you’ll never get to.

Evaluate retrieval quality over time (golden set + citation checks)

If your knowledge base changes weekly and your retrieval doesn’t have tests, you will regress. Quietly. The assistant will still sound confident, and that’s the worst part.

My minimum viable eval setup:

• evals/golden_questions.yml: 25–50 questions that should always be answerable
• evals/expected_sources.yml: expected doc IDs / URLs per question
• A CI job that runs retrieval and checks:
  • Hit rate: did we retrieve at least 1 expected source in top k=5?
  • Citation format: does the answer cite retrieved sources?
  • Refusal behavior: do we avoid answering when retrieval is empty?

Concrete number: start with 25 questions. Get it green. Grow to 100 over time. A 500-question golden set sounds impressive and then slowly turns into a pile of stale YAML.

If you want deeper metrics and failure modes, I’ve got a full playbook in RAG evaluation metrics for retrieval quality and the broader framing in AI engineering evals: regression gates. This post is the starter kit version, not the dissertation.

Publishing the KB without duplicating content

Publishing is where people accidentally fork the truth.

Don’t.

Pick a static docs generator and point it at content/. That’s it.

• MkDocs: dead simple for Markdown-first KBs
• Docusaurus: better if you need versioned docs + React extensions

Concrete number: keep your docs site build under 2 minutes in CI. If it’s slower, it stops being part of the dev loop and becomes “that flaky job that fails later.”

Also: publish the agent layer. Put llms.txt at the root, and link it from your README.

Provenance, licensing, PII, and secrets: the unsexy requirements

This is where most “RAG knowledge base starter kit” repos lie to you by omission.

You’re going to ingest:

• Internal docs containing customer info
• Vendor docs with license constraints
• Chat transcripts with secrets

If you don’t track provenance, you won’t be able to answer basic questions like “can we legally embed this?” or “who owns this page?” You’ll just keep shipping until someone from Legal or Security shows up and ruins your week.

Here’s what I’d enforce:

• Provenance file per page (or frontmatter) with source_url, retrieved_at, license
• PII redaction report per ingestion run (even a JSON file)
• Secrets scanning in CI (pre-commit + PR gate)

If you’re building anything resembling AI security, treat your KB as an attack surface. Indirect prompt injection can arrive through your docs. If you want to go deep on that threat model, start with prompt injection and my prompt injection regression testing in CI.

Concrete number: set a policy that ingestion jobs must run in an environment with zero long-lived credentials. Use OIDC where possible, and keep tokens scoped to read-only.

Turn it into a GitHub template (so others can fork it)

Once your starter kit repo works, make it a template so teammates or the community can generate a new repo with the same structure.

GitHub supports template repositories for exactly this. You mark the repo as a template, and users can click “Use this template” to create a new repository with the same directory structure and files (GitHub Docs).

Practical checklist:

• Replace organization-specific names with placeholders
• Move secrets to ENV.example and document required variables
• Make pipelines/ idempotent (running twice doesn’t duplicate output)
• Add LICENSE and clarify what parts are yours vs ingested

Concrete number: aim for a first-time fork experience that takes 10 minutes to go from “new repo created” to “CI passing.” If it’s longer, adoption drops hard.

One more experience-earned lesson: slug identity is a one-way door. In my own publishing automation, rewriting slugs on live URLs burned 907K impressions of link equity in a single incident. The KB equivalent is renaming canonical doc IDs after other systems depend on them. Pick stable IDs early, then leave them alone.

If you’re building an LLM knowledge base in 2026, the winning move is to stop treating it like documentation and start treating it like software. CI gates, scheduled freshness, and evals aren’t “nice to have.” They’re the only thing standing between a useful KB and a hallucination machine with a clean README.

My prediction: within 12 months, teams that don’t have retrieval evals in CI will be treated the way we treat teams without tests today. Not “immature.” Just not shippable.

If your KB can’t fail a PR, it’s not a system. It’s a folder.

Originally published on kunalganglani.com