The request I hear most often this year sounds the same every time: "We want a bot that answers from our documents and does not make things up." The technical name is RAG, retrieval-augmented generation. Below, without jargon: what it is, who it pays off for, what your company needs to bring, and where these projects break.
RAG in plain language
A language model answers from memory. It read the internet up to some date and knows nothing about your returns policy, September price list or warehouse manual. Ask it directly and it will either admit it does not know or confidently invent something.
RAG changes the order. First the system searches your documents for the passages that relate to the question. Then it hands them to the model with the question and asks it to answer only from them, naming the source document. The model still writes the text, but the source of facts becomes your files, not its memory.
To the user it looks like a chat. To the business it is a way to give people hundreds of pages of documentation without making them read those pages.
The citation in every answer is not decoration. It is the main mechanism of trust.
Why RAG is in 13.6% of postings and vector databases in only 4.5%
In a sample of 903 US AI Engineer postings (Glassdoor, April 2026), RAG is required in 13.6% of ads. That is more often than agents (10.6%) and prompt engineering (8.9%). Vector databases, usually called the heart of RAG, appear in only 4.5%.
The gap between those numbers shows where the work actually is. A database for semantic search can be stood up in a day, and there are plenty of ready options. The hard part comes before and after.
- Documents. Three versions of the same policy in different folders, scanned PDFs, tables that lose their structure when copied.
- Chunking. How to split a document so a rule and its exception do not land in different pieces.
- Permissions. Who is allowed to see which documents, and how to carry that into search.
- Evaluation. How to tell that the bot answers correctly rather than just confidently.
In my review of the AI job market in 2026 I quoted the KORE1 survey: employers name building evaluation systems as the number one skill, not choosing a database. For RAG that is doubly true.
Who it pays off for
Not every company needs a bot on its documents. It pays off where there is a lot of text and many repeated questions about it.
- Support with a big FAQ. Hundreds of articles, agents answering the same questions, first response measured in hours.
- Internal policies and manuals. A new hire spends weeks asking colleagues where things are. HR and accounting answer the same questions about leave and expenses.
- Sales teams with a catalog. Thousands of SKUs, specifications, compatibility, regional terms. A rep digs through ten files while the customer waits.
- Agencies with an archive of proposals. Past pitches, case studies, estimates. A new proposal is assembled from old ones, if they can be found.
The common trait: the answer already exists in the documents, but finding it costs more than asking a person.
Twenty pages of documentation and five questions a day do not need a bot. One FAQ page will do.
What the company needs to bring
I raise this in the first meeting, because it matters more than the choice of model.
- Current documents. The bot answers from what it was given. If a policy is out of date, the bot will confidently relay it, with a precise citation.
- An owner for updates. A named person responsible for keeping the base current. Without one the system degrades within a quarter.
- A list of questions with correct answers. Fifty or a hundred real questions from customers or staff, and what counts as a right answer. Without it nobody can say whether the bot got better or worse after a change.
The third item gets the most resistance and delivers the most value. A couple of days with support logs or email threads is enough to assemble it.
Where these projects break
Four typical failures, each with a defense.
Stale documents. The bot cites last year's price list. Defense: an update owner, and the document date shown in the answer so the user sees how fresh the source is.
Answering outside scope. A customer asks about the legal consequences of terminating a contract, and a bot built on delivery documents tries to answer. Defense: scoped retrieval and a hard rule: if the documents do not contain the answer, say so and hand over to an operator.
Leaking documents. The bot answers a customer with a passage from an internal manual listing cost prices. Defense: permissions enforced at the retrieval level, not in the prompt. The model must never see a document the user may not see.
Hallucinated citations. The model cites a section that does not exist. Defense: citations are generated by the system from the retrieved passages, not written by the model, plus an evaluation set that tracks such cases.
Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027. From what I have seen, document projects die for the same reasons, and almost always nobody built an evaluation set before starting.
A realistic timeline
A pilot on one set of documents takes two weeks. One source (say, the support knowledge base), one scenario (answers for customers or staff), an evaluation set of 50 questions, handoff of hard cases to a person. After two weeks you have a working version to show the team and measure.
Other sources and permissions for several roles come in the second stage. I would not try to cover everything at once: every new source brings its own document quality problems.
How this fits my services
I build this kind of bot under the "AI inside your existing product" service, from 1,200 EUR. That covers RAG, LLM features and integrations into what you already run: a website, a CRM, a support portal, an internal tool. If it is unclear whether the company has suitable documents and a process, start with an AI readiness audit from 450 EUR.
If staff or customers keep asking the same questions about your knowledge base, send me a short brief: what the documents are, who asks, how many questions per week. I will tell you whether it is worth doing and how long it would take.
Originally published at ilinmaks.com.
