Building a Multi-Agent AI Pipeline That Ships: LangGraph, RAG, and Evals That Matter

I spent 18 days building an AI product that converts research papers into audience-tailored PowerPoint presentations. Not a toy — a real deployed thing at doc2slides on Railway that anyone can use.

The interesting parts weren't the "make it work" moments. They were the tradeoffs I had to make honestly, and the times I resisted the temptation to add a "clever" fix that would have made things worse.

This post is about those decisions.

What I built

Doc2Slides takes a PDF and produces a .pptx file tailored to four audiences:

Kid — fun analogies, simple words
Student — educational, terms defined
Engineer — technical depth, assumes domain knowledge
Executive — business focus, impact-oriented

The magic is that the same paper produces radically different output based on the audience. A compiler theory paper for a kid becomes "compilers are like magic helpers." The same paper for an executive becomes "advancing compiler technology with formal frameworks."

Code: github.com/manasviboineypally/doc2slides

The architecture: 5 agents in LangGraph

I built this as a multi-agent pipeline instead of one giant LLM prompt. Here's the flow:

PDF Upload
    ↓
Parser        → extracts sections + metadata
    ↓
Summarizer    → RAG-based section summarization
    ↓
Planner       → designs slide structure for audience
    ↓
Writer        → generates audience-adaptive slide content
    ↓
Builder       → produces editable .pptx file

Each agent is an independent node in a LangGraph state machine. They share a TypedDict state and read/write specific fields.

Here's what the graph definition actually looks like:

from langgraph.graph import StateGraph, END
from app.agents.state import AgentState
from app.agents.parser import parser_agent
from app.agents.summarizer import summarizer_agent
from app.agents.planner import planner_agent
from app.agents.writer import writer_agent
from app.agents.builder import builder_agent

def build_pipeline():
    graph = StateGraph(AgentState)

    graph.add_node("parser", parser_agent)
    graph.add_node("summarizer", summarizer_agent)
    graph.add_node("planner", planner_agent)
    graph.add_node("writer", writer_agent)
    graph.add_node("builder", builder_agent)

    graph.set_entry_point("parser")
    graph.add_edge("parser", "summarizer")
    graph.add_edge("summarizer", "planner")
    graph.add_edge("planner", "writer")
    graph.add_edge("writer", "builder")
    graph.add_edge("builder", END)

    return graph.compile()

Why LangGraph over a sequential chain? Adding a new agent is a 2-line change to the graph. In a sequential chain, adding a new step often means refactoring the previous ones. State-based multi-agent design scales better.

The interesting tradeoff #1: My RAG top-1 precision is 42%

I built an evaluation harness because I wanted to measure quality, not just claim it. Three eval types:

Parser evals — deterministic ground-truth assertions
RAG evals — top-K precision on hand-labeled query→section pairs
Summarizer evals — LLM-as-judge scoring faithfulness, completeness, clarity

The parser evals scored 100% (34/34 checks). The summarizer evals averaged 4.4/5.

But the RAG top-1 precision came in at 42%. Only 3 of 7 queries returned the correct section as the top result.

My first instinct: hide the number. Report top-3 (57%) instead.

What I did instead: publish both numbers and explain why.

Looking at the failures revealed a real limitation of RAG:

Query: "how does the genetic algorithm work?"
Expected section: Methodology
Actual top result: 3.6 Stopping Criteria (a subsection of methodology)

Genetic algorithms are discussed in 6 subsections (3.1 through 3.6). Vector search returns the highest-scoring chunk, not the highest-scoring section. For queries about broad topics, subsections often outrank the parent section because they mention the specific term more densely.

This is a known problem in RAG. Solutions include:

Hierarchical retrieval (search subsections, bubble to parent)
Query rewriting to be more specific
Retrieve top-K and let an LLM pick the right section

None of these are fixed today. But I know exactly what's broken and why — which is more useful than pretending it works.

Lesson: deterministic metrics beat vibes. Vibes let you convince yourself the AI is smart. Metrics tell you where it's dumb.

The interesting tradeoff #2: I refused to use word count as a proxy for content density

Users can request any number of slides between 3 and 50. When the paper's actual content density doesn't match the requested slide count, the LLM either pads shallow sections or compresses dense ones. This creates mild redundancy at high slide counts.

The obvious fix: allocate slides based on section word count. Long section = more slides. Short section = fewer slides.

I almost built this. Then I realized: word count is not content density.

Consider:

A 100-word section with 3 distinct concepts should get multiple slides
A 2000-word section rambling around one idea should get one slide

Word count would systematically reward verbose sections and penalize concise ones. That's not a fix — it's a bug with math.

What I did instead: documented the tradeoff and shipped without the heuristic. From the project's testing_notes.md:

Rejected quick fix: using section word count as a proxy for content density. Word count is not density — a short section may contain multiple distinct ideas while a long section may ramble around one.

Proper solution deferred: content-aware slide allocation with LLM judgment, verified by an evaluation harness that measures output quality against ground truth. Requires infrastructure work not appropriate for the initial version.

Lesson: the right answer to "should I add this heuristic?" is often "no." Heuristics feel like progress. Sometimes they're anti-progress dressed up as pragmatism.

The interesting tradeoff #3: SQLite dev → PostgreSQL prod is one variable

I built with local SQLite during development but deployed to Railway with PostgreSQL. The migration was one line:

# app/db/session.py
DATABASE_URL = os.getenv("DATABASE_URL")
engine = create_engine(DATABASE_URL, echo=False)

For local dev, .env has:

DATABASE_URL=sqlite:///./doc2slides.db

For Railway, the environment variable is:

DATABASE_URL=postgresql+psycopg2://postgres:xxx@host:5432/railway

Nothing else changes. SQLAlchemy models are backend-agnostic.

This is boring engineering. But boring engineering is what lets you sleep at night. When someone asks "how do you handle database migrations?" the answer isn't a clever hack — it's "environment-driven configuration and a repository pattern."

The stack

Layer	Choice	Why
Language	Python 3.13	AI ecosystem
API	FastAPI	Async support, auto Swagger docs
Orchestration	LangGraph	State-based multi-agent
LLM	OpenAI GPT-4o-mini	Cheap enough for iteration, smart enough for structured output
Vector DB	ChromaDB	Local, no cloud dependency
Structured output	JSON mode + Pydantic	Two-layer validation
Database	SQLAlchemy + PostgreSQL	Env-driven, portable
Frontend	Vanilla HTML/CSS/JS	No build step, portable
Deployment	Railway	GitHub CI/CD, managed Postgres

The frontend is worth calling out. I used no framework — just HTML, CSS, and vanilla JavaScript in ~500 lines. Zero build step. Anyone can clone the repo, open the file, and understand it in 5 minutes.

For an MVP, that's a feature, not a limitation.

What I didn't build (and why that's OK)

Skipped:

User authentication
Multi-tenant workspaces
Custom presentation templates
Streaming responses
Job queue with Celery/Redis

Why: MVP. Every feature has a cost. Shipping the core value (PDF → audience-tailored slides) matters more than shipping every possible feature.

For a portfolio project, "I could have added X but chose not to for these reasons" is a stronger answer than "I added X poorly."

Lessons I'd tell my past self

1. Build evals before optimizing. I built the pipeline first, then evals. If I had built evals first, I would have known earlier that my RAG had issues. Now I have to make eval-driven improvements Week 3.

2. Resist heuristics. Every time I thought "this is a quick fix," it was actually a technical debt I was about to bake in. Word count as density. Silent AI slide count overrides. Boolean status flags instead of proper enums.

3. Deploy early. I deployed on Day 16 of 18. I should have deployed on Day 8. Deployment reveals real bugs — environment variable typos, missing dependencies, hardcoded localhost URLs. The sooner you find them, the cheaper they are.

4. Document tradeoffs, not features. Anyone can read code to know what it does. Almost no one leaves notes on why a design choice was made. My testing_notes.md file is where most of the actual engineering thinking lives.