I Built an Agentic AI System From Scratch: RAG, GraphRAG, 12 Tools & Kubernetes

TL;DR: Over five weeks, I built a full-stack Agentic AI system that can plan multi-step tasks, retrieve information using hybrid RAG and GraphRAG, call 12 tools, review its own intermediate results, learn from user feedback, and deploy to Kubernetes.

The project ended up at roughly 6,000 lines of code, 360 tests, and green CI.

More importantly, I built an evaluation harness so I could measure whether the system actually improved instead of just adding more AI components.

GitHub: https://github.com/zda25m005-netizen/agentic-ai-os

Why I Built This

A lot of agent demos follow this pattern:

User → LLM → Tool → Answer

It looks impressive in a demo.

But real-world questions are usually more complicated.

For example:

"Summarize the Q3 business risks from these 40 PDFs, find the products that shipped to Berlin, and explain how those products are connected to the risks."

A single LLM call isn't enough.

The system needs to:

1. Find the right information.
2. Understand relationships between entities.
3. Break the problem into multiple steps.
4. Call different tools.
5. Check whether each step actually worked.
6. Retry when something goes wrong.
7. Produce an answer with citations.
8. Tell me how much the whole operation cost.

So I decided to build the system I would actually want to debug.

Not just an agent demo.

A production-shaped Agentic AI OS.

What I Built

At a high level, the system looks like this:

                    User Goal
                       │
                       ▼
                  ┌──────────┐
                  │  Planner │
                  └────┬─────┘
                       │
                       ▼
                ┌─────────────┐
                │   Executor  │
                └──────┬──────┘
                       │
            ┌──────────┼──────────┐
            ▼          ▼          ▼
          Tools       RAG      GraphRAG
            │          │          │
            └──────────┼──────────┘
                       │
                       ▼
                  ┌─────────┐
                  │  Critic │
                  └────┬────┘
                       │
                ┌──────┴──────┐
                │             │
             APPROVE        RETRY
                │             │
                ▼             └──→ Executor
             Finalize
                │
                ▼
       Answer + Trace + Metrics

The backend is built with FastAPI + LangGraph, with a Next.js frontend.

For retrieval, I use both:

• Hybrid RAG
• GraphRAG with Neo4j

The system currently exposes 12 tools, including:

• Python execution
• SQL
• Web search
• RAG search
• Graph search
• HTTP requests
• File operations

Each tool is isolated and guarded according to its use case.

The Part Most Agent Demos Skip: Evaluation

This is probably the part I'm most proud of.

It is very easy to build an AI system that looks better.

It is much harder to prove that it actually is better.

So I built the evaluation system alongside the product.

The harness measures things like:

• Recall@k — did the correct source appear in the top-k results?
• LLM-judge correctness
• Citation accuracy
• Agent task-success rate
• GraphRAG fact coverage
• Retrieval performance across different strategies
• Reranker improvements
• Fine-tuning before/after comparisons

I also included a retrieval ablation:

Vector Search
      ↓
BM25
      ↓
Hybrid RAG
      ↓
Hybrid + Reranker

This matters because I don't want to simply say:

"Hybrid RAG is better."

I want to be able to show:

"Here are the test cases, here is the baseline, here is the hybrid result, and here is what changed."

And when something doesn't improve the baseline, the evaluation reports that too.

I think honesty in evaluation is a feature.

1. Hybrid RAG: Why Use Two Retrieval Strategies?

Dense vector search is excellent at understanding semantic meaning.

For example:

"How do I reset my password?"

can retrieve:

"Steps for recovering access to your account"

even though the wording is different.

But semantic search can struggle with exact identifiers.

Imagine searching for:

SKU-4471

That's where BM25 can be much better.

So I combined both approaches.

The system performs:

        Query
          │
     ┌────┴────┐
     ▼         ▼
 Vector       BM25
 Search       Search
     │         │
     └────┬────┘
          ▼
 Reciprocal Rank
     Fusion
          │
          ▼
      Reranker
          │
          ▼
      Top Results

For fusion I use Reciprocal Rank Fusion (RRF).

Instead of combining the raw scores from vector search and BM25, RRF combines their rankings.

The basic idea is:

RRF score(d) = Σ 1 / (k + rank(d))

where:

• rank(d) is the position of document d in a result list.
• k controls how quickly the contribution decreases.
• I use k = 60.

For example, if a document appears at rank 1 in one result list and rank 4 in another:

RRF score = 1 / (60 + 1) + 1 / (60 + 4)

The important part is that RRF uses rank rather than raw retrieval scores.

This makes it useful when combining vector search and BM25 because their raw scores don't necessarily live on the same scale.

The actual implementation is tiny:

def reciprocal_rank_fusion(result_lists, k=60, limit=5):
    scores, payloads = {}, {}

    for hits in result_lists:
        for rank, hit in enumerate(hits, start=1):
            scores[hit.id] = (
                scores.get(hit.id, 0.0)
                + 1.0 / (k + rank)
            )
            payloads.setdefault(hit.id, hit.payload)

    ranked = sorted(
        scores.items(),
        key=lambda kv: kv[1],
        reverse=True
    )

    return [
        SearchHit(
            id=i,
            score=s,
            payload=payloads[i]
        )
        for i, s in ranked[:limit]
    ]

I intentionally kept the implementation small instead of hiding the ranking logic behind another abstraction.

That makes the behavior easy to understand, test, and debug.

2. GraphRAG: When Documents Aren't Enough

Traditional RAG is great at retrieving relevant passages.

But some questions are fundamentally about relationships.

For example:

"How is Product A connected to Customer B?"

or:

"Which products are associated with the risks mentioned in this document?"

Finding individual passages isn't necessarily enough.

So I added a knowledge graph.

The ingestion pipeline looks roughly like this:

Documents
    │
    ▼
Chunks
    │
    ▼
LLM Entity + Relationship Extraction
    │
    ▼
Neo4j Knowledge Graph

Entities and relationships are extracted from the documents and stored in Neo4j.

The graph writes use MERGE, making ingestion idempotent.

At query time:

User Query
    │
    ▼
Entity Detection
    │
    ▼
k-hop Neighborhood
    │
    ▼
Relevant Graph Facts

The graph results are then combined with the normal RAG passages.

The API supports:

/ask?mode=vector
/ask?mode=graph
/ask?mode=fused

I also expose GraphRAG through a dedicated:

graph_search

tool.

This gives the agent another way to answer questions where relationships matter more than isolated text passages.

3. What Actually Makes It an Agent?

This was an important design question for me.

Calling a tool doesn't automatically make something an agent.

The interesting part is the control loop.

After the executor completes a step, a Critic evaluates the result.

Conceptually:

Step
 │
 ▼
Execute
 │
 ▼
Critic
 │
 ├── APPROVE ──→ Next Step
 │
 └── RETRY ────→ Execute Again

The Critic can return:

APPROVE

or:

RETRY: The retrieved evidence does not support the claim.

The retry mechanism is intentionally bounded.

async def review(step, result, chat_fn):
    raw = (
        await chat_fn(
            _critic_messages(step, result)
        )
    ).strip()

    if raw.lower().startswith("approve"):
        return APPROVE, ""

    reason = (
        raw.split(":", 1)[1].strip()
        if ":" in raw
        else raw
    )

    return RETRY, reason

And at the graph level:

if verdict == RETRY and retries < MAX_RETRIES:
    return {
        "cursor": idx,
        "retries": retries + 1,
        ...
    }

That last condition is important.

The loop must terminate.

Agent systems can become expensive very quickly when they are allowed to keep reasoning forever.

So I explicitly enforce retry limits.

The goal is simple:

No unbounded loops. No runaway bills.

4. The 12-Tool Execution Loop

The executor runs a function-calling loop over 12 tools.

Some examples include:

Python
SQL
Web Search
RAG
Graph Search
HTTP
Files
...

I also added safeguards around the tools.

For example:

• Python execution is sandboxed.
• SQL is read-only.
• HTTP requests have SSRF protections.
• File access is path-guarded.

Another design decision I made:

A tool failure should not automatically kill the entire agent run.

Instead of:

Tool Error → Crash

the system returns the error to the model:

Tool Error
    │
    ▼
Agent observes error
    │
    ▼
Agent decides what to do next

That gives the agent an opportunity to recover when possible.

It also makes debugging much easier because the error becomes part of the execution trace.

5. Learning From User Feedback

The system also has a feedback loop.

Users can give:

👍
👎

and optionally provide a better answer.

That feedback is used in two ways.

Learned Reranking

The feedback can train a lightweight reranker.

When there isn't enough feedback yet, the system falls back to the LLM reranker.

That gives me:

Cold Start
    ↓
LLM Reranker
    ↓
User Feedback
    ↓
Learned Reranker

rather than deploying a learned model before there is enough data.

DPO Preference Pairs

I also export preference data as JSONL.

Conceptually:

Chosen Answer
      vs
Rejected Answer

I intentionally describe this as DPO, not RLHF.

There is:

• No reward model
• No online reinforcement learning

The goal is to keep the terminology technically accurate.

6. Observability: Because "It Feels Slow" Isn't a Metric

One of my biggest goals was to make the system observable.

The API exposes Prometheus metrics for:

• Request count
• Latency
• Token usage
• Estimated USD cost
• Agent-node execution
• Tool execution
• Errors

The metrics are visualized in Grafana.

So instead of asking:

"Why is this agent expensive?"

I can look at the cost of individual parts of a run.

For example:

Request
  │
  ├── Planner       $0.002
  ├── RAG           $0.000
  ├── Tool #4       $0.001
  ├── Critic        $0.003
  └── Finalizer     $0.002
                  --------
                    $0.008

I also added:

• Optional Langfuse traces
• /readyz dependency checks
• Structured JSON logs
• Request IDs

The result is that an agent run isn't just an answer.

It's an answer + execution trace + metrics.

7. Kubernetes Deployment

The project isn't just:

docker compose up

I also created a Helm deployment.

The chart includes:

• Ingress
• API autoscaling
• ConfigMaps
• Secrets
• Non-root containers
• NetworkPolicies

The CI pipeline creates a kind Kubernetes cluster and verifies the deployment.

It checks the health endpoint after deployment.

So the deployment isn't merely:

"Here's a Helm chart. It should work."

The CI actually exercises it.

That distinction matters.

A deployment configuration that looks correct is not the same thing as a deployment that has actually been tested.

8. The Evaluation Harness

The evaluation system is what ties the whole project together.

I didn't want to evaluate the system only by asking:

"Does the answer look good?"

Instead, I separated evaluation into multiple dimensions.

Retrieval

I measure:

• Recall@k
• Vector-only retrieval
• BM25 retrieval
• Hybrid retrieval
• Hybrid + reranker

This makes it possible to see which retrieval strategy actually helps.

Answer Quality

I measure:

• LLM-judge correctness
• Citation accuracy
• Fact coverage

Agent Performance

For multi-step tasks, I measure:

• Task-success rate
• Successful tool execution
• Retry behavior
• Final answer quality

GraphRAG

For graph-based questions, I measure:

• Relevant entity retrieval
• Relationship coverage
• Fact coverage

Most importantly, the evaluation reports the sample size.

I don't want to publish:

"Our approach improved accuracy by 17%."

without also saying:

"across N evaluation examples."

Numbers without context can be misleading.

What the Five Weeks Taught Me

1. Measure From Day One

The evaluation harness ended up being one of the most valuable parts of the project.

Without it, I would have been guessing whether changes actually improved the system.

Every new component creates another question:

"Did this actually help?"

An evaluation gives you an answer.

2. More AI Isn't Automatically Better

Adding:

RAG
+
GraphRAG
+
Reranker
+
Critic
+
12 Tools

sounds impressive.

But every additional component introduces:

• Latency
• Cost
• Failure modes
• Complexity

The evaluation needs to justify the complexity.

Sometimes the simplest solution wins.

3. Bound Your Agent Loops

This is one of the easiest things to overlook.

An agent that can endlessly retry is not robust.

It's an expensive bug.

Set explicit limits.

Track the cost.

Make the failure mode predictable.

4. Don't Hide Bad Results

One of the most useful principles I learned from this project:

A benchmark that tells you your idea didn't work is still a successful benchmark.

If a reranker doesn't beat the baseline, report it.

If GraphRAG doesn't help a particular query type, report it.

If a model performs worse after fine-tuning, report it.

That information is much more valuable than a perfect-looking README.

5. Keep Infrastructure Boring

I also learned a less exciting but very practical lesson.

Small infrastructure mistakes can waste hours.

For example:

docker compose up -d --build

became part of my development routine.

At one point, I spent an afternoon debugging behavior that turned out to be a stale Docker image running old code.

The lesson:

When the behavior makes no sense, check what code you're actually running.

The Result

After five weeks, the system looks roughly like this:

But the important part isn't the number of technologies.

It's that each component exists for a reason and has a test or evaluation behind it.

Architecture at a Glance

Putting everything together:

                         ┌──────────────┐
                         │     User     │
                         └──────┬───────┘
                                │
                                ▼
                         ┌──────────────┐
                         │    Planner   │
                         └──────┬───────┘
                                │
                                ▼
                    ┌──────────────────────┐
                    │       Executor       │
                    │   Function Calling   │
                    └──────────┬───────────┘
                               │
          ┌────────────────────┼────────────────────┐
          │                    │                    │
          ▼                    ▼                    ▼
     ┌─────────┐         ┌──────────┐        ┌───────────┐
     │  Tools  │         │ Hybrid   │        │ GraphRAG  │
     │   ×12   │         │   RAG    │        │  Neo4j    │
     └─────────┘         └────┬─────┘        └─────┬─────┘
                               │                    │
                               └─────────┬──────────┘
                                         │
                                         ▼
                                  ┌──────────────┐
                                  │    Critic    │
                                  └──────┬───────┘
                                         │
                                  ┌──────┴──────┐
                                  │             │
                               APPROVE        RETRY
                                  │             │
                                  ▼             │
                              Finalize ◄────────┘
                                  │
                                  ▼
                      ┌─────────────────────────┐
                      │ Answer + Citations      │
                      │ Execution Trace         │
                      │ Tokens + Cost           │
                      │ Evaluation Metrics      │
                      └─────────────────────────┘

Around the system sits the infrastructure layer:

                 ┌─────────────────────────┐
                 │      Observability       │
                 │ Prometheus + Grafana     │
                 │ Optional Langfuse        │
                 └─────────────────────────┘

                 ┌─────────────────────────┐
                 │       Deployment        │
                 │ Docker + Helm + K8s     │
                 │ kind CI verification    │
                 └─────────────────────────┘

What I'd Build Next

There are still several things I'd like to improve.

Better Evaluation Datasets

The evaluation harness is only as good as the questions being evaluated.

A larger, more diverse benchmark would make the results more meaningful.

Better Agent Cost Optimization

The Critic and multiple retrieval stages add latency and token usage.

A future version could dynamically decide when a review step is actually necessary.

More Deterministic Tool Policies

Some tools can be made more deterministic by moving safety and validation logic outside the LLM.

The model should decide what to do.

The infrastructure should decide what is allowed.

Better Long-Term Memory

The current feedback system is intentionally lightweight.

A stronger memory layer could learn user preferences and task patterns while still keeping the system auditable.

Try It Yourself

The project is available here:

GitHub: https://github.com/zda25m005-netizen/agentic-ai-os

Clone the repository and run:

docker compose up --build

Then open:

Frontend → http://localhost:3000
API      → http://localhost:8000

The repository contains:

• Architecture documentation
• Setup instructions
• Evaluation harness
• Retrieval experiments
• GraphRAG implementation
• Agent orchestration
• Tool implementations
• Observability configuration
• Docker configuration
• Helm deployment
• Kubernetes CI verification

Final Thoughts

I started this project wanting to understand what it takes to move beyond a simple:

LLM + Prompt

and build something closer to a real AI system.

The biggest lesson wasn't GraphRAG.

It wasn't LangGraph.

It wasn't even the multi-agent architecture.

It was measurement.

Building an agent is relatively easy.

Building an agent where you can answer:

"Did this change actually make the system better?"

is much harder.

That's the part I'm continuing to work on.

If you're building agentic systems too, I'd especially love feedback on the evaluation methodology, the retrieval experiments, and the trade-offs in the architecture.

Thanks for reading.