I spent 6 weeks building RAG Sentinel – an open-source evaluation tool for RAG systems.
The problem: RAGAS scored my RAG system 0.91. But it was still hallucinating citations and ignoring half the retrieved context.
RAGAS doesn't answer the questions I actually cared about:
→ Does the answer actually USE the retrieved documents?
→ Do the citations exist in the source documents?
→ Does the answer contradict its own sources?
→ Is the system's confidence actually accurate?
So I built 5 custom metrics to measure exactly these.
What I Built
RAG Sentinel is a production-ready evaluation platform that goes beyond RAGAS with:
🔹 Retrieval-Generation Alignment – Does the answer actually use the retrieved docs? RAGAS faithfulness can score 0.9 even if half the context is ignored.
🔹 Citation Accuracy – Do citations actually exist in the source documents? Or are they hallucinated?
🔹 Context Contradiction – Does the answer contradict its own sources? Standard metrics don't check for this.
🔹 Confidence Calibration – When the system says "90% sure," is it actually right 90% of the time?
🔹 Production Monitoring Dashboard – Track quality trends. Detect degradation before users notice.
Tech Stack
| Layer | Technology |
|---|---|
| Backend | FastAPI (Python) |
| Frontend | React (Netlify) |
| LLM | Groq (openai/gpt-oss-120b) |
| Deployment | Railway (backend) + Netlify (frontend) |
The Hard Parts
The code took ~2 weeks. Getting it deployed took ~4.
- CORS preflight errors: 3 days
- PostgreSQL misconfiguration: 2 days
- LLM API deprecated mid-project: Rewrote the evaluation pipeline entirely
That's where I learned the real difference between "works on my laptop" and "someone can actually use it."
What I Learned
🔸 Building systems > building chatbots. Anyone can call an API. Building a monitoring platform is harder but way more valuable.
🔸 Deployment is 50% of the work. The code is the easy part. Making it work in production is where the real learning happens.
🔸 Ship. Then iterate. I deployed with basic features, added more as I went. If I'd waited for "perfection," I'd still be coding.
Links
🔗Linked In:https://www.linkedin.com/feed/update/urn:li:activity:7495058202019938307/
🔗 Live Demo: https://rag-sentinel.netlify.app
🔗 GitHub: https://github.com/Abdullah-Ahmad-2025/rag-sentinel
Built solo. 1st-year BS AI student. 6 weeks.
If you're building RAG systems – I'd genuinely love your feedback. What metrics do you actually care about?
