Most RAG chatbots that disappoint were not failed by the model. They were failed at retrieval. I wrote a fuller walkthrough of the pipeline here (RAG Chatbot Development), and this is the opinionated version of where they actually break.
The pipeline is six steps and only two of them get attention
Ingest, chunk, embed, store, retrieve, generate. Almost everyone spends their energy on the last step, tuning the prompt and swapping models, and on picking a vector database for the store step. The quality of the answers is actually decided at chunk and retrieve, before the model ever runs. If those two are wrong, no amount of prompt engineering saves you.
Chunking is the quiet decision that sets your ceiling
Bad chunking gives you the right document and the wrong passage, or a passage that does not actually contain the answer. Split by structure and meaning rather than a blind character count, keep some overlap so answers are not sliced in half, and preserve headings so a chunk still knows what section it came from. This unglamorous step sets the ceiling on everything downstream.
Pick the vector database for where your data must live, not for benchmarks
Qdrant, Pinecone, and pgvector will all do similarity search fine. The real question is operational. Self-host with Qdrant or pgvector when private data has to stay inside your own infrastructure, and reach for a managed option when you simply do not want to run the thing. Similarity search is commoditized; data residency and who carries the ops burden are the actual differentiators.
RAG reduces hallucination, it does not delete it
Retrieved context makes the model more grounded, but it will still fill a gap if you let it. Require citations, instruct it to say it does not know when the retrieved context is thin, and surface the sources so a human can verify the claim. A confident answer with no traceable source is the failure mode users remember.
Production adds the two things demos always skip
Per-user permissions so retrieval can never return a document the current user is not allowed to see, and scheduled re-indexing so answers do not quietly go stale as the source material changes. These are the two places a slick prototype fails to become a real product.
The short version
Get retrieval right and an average model gives good, sourced answers. Get it wrong and the best model on the market will confidently invent things. The full pipeline breakdown, ingest through generate, is here: RAG Chatbot Development: How It Works.
This article was drafted with AI assistance (Claude) and reviewed and edited by me.
