Fine-Tuning vs RAG vs Prompting: What Each Costs

The fine-tuning vs RAG question usually arrives as a statement: "we need to fine-tune a model on our data." It is one of the most expensive sentences in enterprise AI, and it is usually wrong. Not always, but usually. The request nearly always means one of two things: the model does not know about our business, or the model does not answer the way we want. Fine-tuning is a poor solution to the first and an expensive solution to the second.

Choosing between fine-tuning, RAG and prompting is not a technical preference. Each fixes a different category of problem, and picking the wrong one produces months of work that does not address the complaint.

The rule that saves the most money: If the problem is that the model does not know something, use retrieval. If the problem is that the model knows but answers in the wrong style, format or length, improve the prompt first and consider fine-tuning only if that fails. Fine-tuning teaches behaviour, not facts, and teams that use it to inject knowledge get a model that is confidently wrong in a new house style.

Fine-Tuning vs RAG vs Prompting: What Each Does

The distinction is simpler than the discourse suggests.

Prompting changes the instructions you send with each request. It shapes tone, format, structure and reasoning approach, and it can supply a modest amount of context inline. It takes effect immediately, costs nothing beyond the tokens, and can be changed in production in seconds.

Retrieval fetches relevant material from your own content at request time and places it in the context before the model answers. It gives the model access to information it was never trained on, including things that changed this morning. The model itself is unchanged; you are improving what it is given to work with. Our explainer on retrieval-augmented generation covers the mechanics.

Fine-tuning adjusts the model's weights using examples of the behaviour you want. It is genuinely effective at teaching consistent format, tone and task-specific patterns that are hard to describe in words but easy to demonstrate. It is poor at teaching facts, because facts learned this way cannot be updated, audited or cited, and the model has no way to tell you when it is drawing on them incorrectly.

The confusion arises because all three change the output. Only retrieval changes what the model knows.

What Each Costs

Figures reflect typical UK delivery for a mid-sized business application.

Prompting. Days rather than weeks, typically £1,000 to £5,000 including the evaluation set you should build alongside it. Running cost is whatever the tokens cost, and a longer system prompt raises it slightly, which prompt caching largely offsets.

Retrieval. Four to twelve weeks, commonly £15,000 to £60,000 depending on how messy the source content is. The build cost sits mostly in ingestion: getting documents out of the systems that hold them, chunking them sensibly, handling permissions so users only retrieve what they may see, and keeping the index current. Running cost adds vector storage and a slightly larger context on every request.

Fine-tuning. Two to eight weeks of engineering plus the dataset, commonly £20,000 to £80,000. The training compute is usually the smallest line; the cost is in producing several hundred to several thousand high-quality examples, which is human work by people who know the domain. Running cost may be higher too, since a fine-tuned model often carries a premium or requires dedicated hosting.

The asymmetry is the point. Prompting is cheap enough to try first even if you are fairly sure it will not be enough, because you will need the evaluation harness regardless and you will learn what the actual failure mode is.

The Order to Try Them In

Working through these in sequence costs less than jumping to the end, even when you end up at the end anyway.

Start with the evaluation set. Thirty to a hundred real inputs with known-good outputs. Without this you cannot tell whether any change helped, and every subsequent decision is guesswork. This is the same harness described in our OpenAI API integration guide.

Then improve the prompt. Be specific about format, give a few examples of good output inline, state what to do when the answer is not known. A surprising proportion of "the model is not good enough" complaints resolve here, particularly around verbosity and structure.

Then try a larger or different model. Frequently cheaper than any customisation work and takes an afternoon to evaluate. Our guide to moving off OpenAI covers running that comparison properly.

Then add retrieval, if the failure is knowledge. If the model is answering questions about your products, policies or documents and getting them wrong or refusing, this is the step that fixes it.

Then consider fine-tuning, if the failure is behaviour. By now you have an evaluation set, a tuned prompt and, if relevant, a retrieval pipeline. If output still does not match what you need in style or structure, and you can demonstrate the desired behaviour in hundreds of examples more easily than describe it, fine-tuning is the right tool.

Most projects stop at step three or four. That is the saving.

When Fine-Tuning Genuinely Wins

There are real cases, and dismissing them entirely would be as wrong as reaching for it first.

Consistent structured output at scale. When every response must follow a rigid format that is tedious to specify in a prompt, a fine-tuned model produces it more reliably and with far fewer input tokens, which pays for itself at volume.

A specialised tone or domain register. Legal drafting, clinical notes, regulated financial communication: styles with strong conventions that practitioners recognise immediately and that are difficult to capture in instructions.

Classification with subtle, hard-to-articulate boundaries. When your team can label examples consistently but cannot explain the rule, that is exactly what training on examples is for.

Cost reduction at high volume. A smaller fine-tuned model matching a larger general model on one narrow task can cut per-request cost substantially. This only makes sense above a volume where the saving exceeds the build and maintenance cost.

The common thread is behaviour, not knowledge. If you can state what you want in a paragraph, prompt it. If you can only show it, train it.

The Costs Nobody Quotes

Three ongoing burdens are routinely left out of fine-tuning business cases.

The dataset ages. Your fine-tuned model reflects the examples it saw. When products, policies or house style change, the examples are stale and the model keeps producing the old behaviour confidently. Budget for periodic retraining as a standing cost, not a one-off.

You are pinned to a base model. Fine-tuning attaches to a specific version. When that base is deprecated or a better model arrives, you retrain to move, which is a real switching cost that a prompt does not have.

Evaluation becomes mandatory rather than optional. With prompting you can eyeball a regression. With a fine-tuned model you cannot inspect why behaviour changed, so the evaluation harness becomes the only instrument you have.

Retrieval has a smaller equivalent: the index must stay current, permissions must stay correct as people change roles, and someone must notice when ingestion silently stops. Cheaper than retraining, but not free.

Get the Diagnosis Before the Build

Mecanik builds retrieval systems, evaluation harnesses and fine-tuning pipelines as part of our AI integration services, and we start by establishing which of the three problems you actually have.

That diagnosis is usually a short engagement and it frequently ends with a recommendation that costs you far less than the project you were planning. Where fine-tuning is genuinely the right answer we will say so and scope the dataset work honestly, because that is the part that determines whether it succeeds. For the wider budget picture, our AI integration cost guide sets out build and running costs separately.

Tell us what the model is getting wrong, in the words your users would use, and we will tell you which of these three it is.

Frequently Asked Questions

Should I fine-tune a model on my company data?
Usually not. Fine-tuning teaches behaviour rather than facts, and knowledge learned that way cannot be updated, cited or audited. If the problem is that the model does not know about your products, policies or documents, retrieval is the correct approach and is cheaper to build and maintain.

What is the difference between fine-tuning and RAG?
Retrieval fetches relevant material from your content at request time and places it in the context, so the model can answer from information it never saw in training. Fine-tuning adjusts the model's weights using examples, changing how it behaves rather than what it knows.

How much does fine-tuning cost?
Typically £20,000 to £80,000 for a mid-sized business application, with two to eight weeks of engineering. Training compute is usually the smallest component; most of the cost is producing several hundred to several thousand high-quality examples, which requires people who know the domain.

How much does a RAG system cost to build?
Commonly £15,000 to £60,000 over four to twelve weeks, driven mainly by how messy the source content is. Most of the effort goes into ingestion, chunking, handling permissions so users retrieve only what they may see, and keeping the index current.

In what order should I try these approaches?
Build an evaluation set first, then improve the prompt, then try a larger or different model, then add retrieval if the failure is knowledge-related, and only then consider fine-tuning if the failure is behavioural. Most projects resolve before the final step.