I'm a Java backend engineer. Three years of Spring Boot, Kafka and Postgres in a regulated healthcare environment, and minimal ML background to speak of.
On deciding that I should learn retrieval-augmented generation properly, all the tutorials I came across recommended that I use Python and LangChain. Since I didn't want to swap the technology stack that I am familiar with for one that was just being used as an example, I based my implementation on the JVM: using Java 21, Spring Boot, Spring AI, and PostgreSQL together with pgvector.
Upload a PDF or a DOCX file; it will be divided into chunks and then inserted into the system. When you pose a question, the question is also embedded, after which cosine similarity is used to identify the most similar chunks, and these chunks are included in the prompt as context, the model then responds based on this context rather than drawing on its training data. There are two services involved, with two separate endpoints. The time taken goes from a weekend to just working.
By that stage it had been completed and was even ready to be demonstrated.
Yet I had no way of knowing whether it was of any value, and the fact that 'the answers look right when I try it' isn't a proper measure. That's why, before adding anything else, I constructed an evaluation harness.
The setup
A fixed 20-question set against a single ingested document, my own CV:
12 single-passage questions — answerable from one chunk
5 multi-passage — needing two or more
3 unanswerable — things the document genuinely doesn't contain
The criteria are retrieval accuracy (to determine whether the right chunk was retrieved), answer correctness, and the number of times the system properly refused to answer questions that could not be answered.
The most important of all these criteria in an actual deployment is that one; a system which confidently answers a question for which it has no basis is worse than one which returns no answer.
Run 1: the baseline
The chunk size is 800, which is the default setting for Spring AI, and the similarity threshold is 0.35.
Metric Result
Retrieval accuracy (n=17) 59%
Answer correctness (n=17) 47%
Unanswerable correctly declined 3 of 3
The reason was simple. Since a one-page CV consists of only a few hundred words, and the chunk size is 800, the whole document was divided into just two chunks. Each query, no matter how specific it was, was then matched against the same two large blocks of text. Thus, there was nothing for the similarity search to distinguish between.
Run 2: the obvious fix
Reduce the chunk size to 150. The threshold remains unchanged.
Metric Result
Retrieval accuracy (n=17) 76% (+17pp)
Answer correctness (n=17) 59% (+12pp)
Unanswerable correctly declined 2 of 3
Retrieval is up by seventeen points. Correctness has increased by twelve. Refusals have gone down.
That is why I have written this post. If I had merely monitored the headline metrics, I would have treated this as a success.
What actually happened
When I looked again at the per-question scores, the result was even worse than what a regression would show.
The threshold had always been completely inactive.
Each of the three unanswerable questions had a similarity score ranging from 0.44 to 0.46 when compared with their best chunk; the threshold set was 0.35 and all of them exceeded it. The system had always been retrieving actual context for questions to which the document could not provide an answer and had then depended on the model to spontaneously say 'I don't know'.
The solution was to use two rather rough chunks—fortunately those worked—since the retrieved context was too general for the model to build a reasonable answer from.
It gave way when broken down into finer, more thematically distinct sections. When I asked about the load-test response time for one of my projects—even though that figure does not in fact apply to that project—the system took the '2–3 ms' value from a similar project in a nearby section and made up an answer, assigning it to the wrong project.
Better retrieval made the hallucination seem more believable.
The obvious fix doesn't work either
It is certain that the threshold should be raised; if it is pushed above 0.46 then the unanswerable questions will be filtered out.
I arranged all twenty questions according to the similarity score of their top chunk; the three unanswerable ones—with scores of 0.440, 0.463, and 0.464—are located within the answerable range, this range extending from 0.423 to 0.629.
A question that was legitimately answerable received a score of 0.423, which was lower than that of each of the three unanswerable ones.
There isn't a cutoff point which divides the two groups; any threshold high enough to pick up the bad questions will also reject the good ones. This isn't a parameter that needs adjusting; it's the case of two distributions overlapping.
Can this be applied in general?
The entire procedure was carried out using a one-page document; it is fair to ask if the tuning had any significance beyond that particular corpus, so I applied the same method to a 72-page MSc dissertation which also had its own 20-question set, with the same 12/5/3 split, in a clean single-document store.
Same config as the CV — chunk size 150, top-k 4:
Metric Result
Retrieval accuracy 59%
Answer correctness 71%
Unanswerable declined 3 of 3
It reverted to the baseline. The reason being that a top-k value of 4 retrieves more than half of the 7-chunk CV for every query, but only about 2% of the 199 chunks of this document. Since the document is long and repeats its main ideas in the Background, Methods, Results and Discussion sections, several chunks that are almost identical and come from the wrong chapter compete for the top positions rather than the one chunk which actually contains the fact in question.
Raising top-k to 8:
Metric Result
Retrieval accuracy 71% (+12pp)
Answer correctness 76% (+5pp)
Unanswerable declined 3 of 3
It is better, but for five of the twenty questions retrieval was still not achieved—in these cases the correct chunk is not even among the top 8 according to cosine similarity, and this problem cannot be solved by any amount of top-k tuning.
The degree of similarity remained the same as you would expect, since top-k has no effect on the individual chunk scores. Unanswerable questions obtained scores as high as 0.620 in this case, which is above the 0.448 minimum score for answerable questions.
What I took from it
The chunk size and the top-k value both need to be adjusted as the size of the corpus increases. A configuration that has been tuned using a one-page document will be incorrect when applied to a seventy-two page document. There is no single default setting that works in all cases, so it's important to be aware of this point before copying one from a tutorial.
The cosine similarity based on a single vector cannot tell the difference between "this text is about what you asked" and "this text answers what you asked"; both of these expressions receive a high score. This is the real limitation, and it is for this reason that the threshold approach fails rather than requiring calibration.
The fact that two metrics are moving in opposite directions is noteworthy. If I had only looked at retrieval accuracy, I would have regarded run 2 as a clear improvement and then moved on to a system that had been gradually becoming better at hallucinating.
The solution involves hybrid search — that is, combining keyword matching with vector similarity — or including a reranking stage among the retrieved candidates. This is something that is planned. However, I prefer to include it only after having gathered solid evidence to support it, rather than just because a blog post suggested it.
The wider point
It took a weekend to build the RAG loop, but it was a lot longer before I realised it wasn't working the way I had expected, and that experience taught me more.
It was never the model that posed the difficulty. The engineering work lies in the chunking strategy, in the quality of retrieval, and in knowing when the proper answer is 'I don't know'. This is rather reassuring if you are a backend engineer unsure about whether you can make any contribution in this area. For the most part, these issues are data and retrieval problems disguised as AI problems, and the kind of discipline that ensures their reliability is the same discipline that ensures the reliability of any system.
The complete code, the full methodology, and the raw results for each question are available at github.com/amritpalkhajuria/ask-my-docs
The application was developed using Java 21, Spring Boot, Spring AI, and PostgreSQL along with pgvector. There are 26 tests in the continuous integration process, and none of these tests make use of a paid API.
