RAG or fine-tuning?
Almost every project that asks for fine-tuning needs retrieval. The distinction is knowledge versus behaviour, and confusing them is expensive.
Use RAG when the model needs to know your facts — documents, policies, product data. Use fine-tuning when it needs to behave a certain way — a format, a tone, a classification it keeps getting wrong. Most business problems are knowledge problems, which is why most projects that ask for fine-tuning actually need retrieval.
Updated August 2026
Short answer
Start with retrieval. Fine-tune only when you can point at behaviour that retrieval cannot fix.
Knowledge or behaviour
The clearest way to hold this: RAG changes what the model knows, fine-tuning changes how it acts. A model that cannot answer questions about your refund policy has a knowledge problem. A model that answers correctly but in five rambling paragraphs when you need three fields of JSON has a behaviour problem.
Knowledge problems are far more common, and retrieval solves them without retraining anything. Your documents go into a vector store, the relevant pieces are found at question time and handed to the model as context. Update a policy and the system knows it immediately — no retraining, no waiting.
Fine-tuning bakes patterns into the weights. It is genuinely good at format and tone, and at narrow classification where a smaller cheaper model can be taught to match a larger one. It is bad at facts: a fine-tuned model will state outdated information with total confidence, because that is what it was taught.
The expensive mistake is fine-tuning on a knowledge problem. You spend weeks preparing data, produce a model that sounds authoritative and is wrong, and have no way to correct it short of retraining. We have been called in to unpick exactly that more than once.
Side by side.
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Facts, documents, policies, product data | Format, tone, narrow classification |
| Updating | Change the document, done | Retrain the model |
| Time to first result | Days | Weeks, including data preparation |
| When wrong | Traceable — you can see what it retrieved | Opaque — the answer came from the weights |
| Cost driver | Retrieval infrastructure and tokens | Training runs and labelled data |
| Handles change | Immediately | Only at the next training run |
| Common mistake | Chunking documents badly | Using it to teach facts |
Choose RAG when
- The model needs to know things it was never trained on
- The information changes and staleness would be a problem
- You need to show where an answer came from
- You want something working in weeks rather than months
Choose fine-tuning when
- Output format or tone is the problem, not correctness
- A narrow classification task keeps failing despite good prompts
- A smaller tuned model could replace an expensive frontier one at volume
- The behaviour you want cannot be described in a prompt, only demonstrated
Where projects go wrong
Fine-tuning to teach facts
The most common and most costly error. The model learns to sound like your documents rather than to know them, and every correction means another training run.
Skipping the evaluation set
Without a set of questions with known-good answers, there is no way to tell whether a change helped. The first thing worth building on any AI project is the harness, not the prompt.
Vector search alone
Embeddings miss exact matches — product codes, names, error strings. Keyword and vector search together outperform either, and the gap is not subtle.
Ignoring how documents are chunked
Retrieval quality is decided more by chunking than by model choice. A table split across two chunks answers nothing, and no amount of prompt engineering rescues it.
Questions that resolve it
The first is usually enough.
If you pasted the right document into the prompt, would the answer be correct?
If yes, you have a knowledge problem and retrieval is the answer. This single question settles most projects.
How often does the underlying information change?
Anything changing monthly or faster rules out fine-tuning on practical grounds — you cannot retrain that often, and stale weights state old facts confidently.
Is the problem the answer, or the shape of the answer?
Wrong answers point at retrieval. Correct answers in the wrong format, tone or structure point at fine-tuning, and that is a genuine case for it.
Do you need a smaller model to match a larger one?
This is fine-tuning's strongest case. A tuned small model matching a frontier model on one narrow task can cut cost substantially at volume.
What people ask next.
Yes, and mature systems often do — retrieval for knowledge, a light tune for format. Doing both from the start is usually premature; get retrieval right first, then tune only if a specific behaviour still resists.
An evaluation set: questions with known-good answers, scored on every change. Without it you are guessing, and every change to the system is a coin flip. It is the first thing we build.
That depends on the model. Hosted APIs mean data leaves. Open-weight models on your own servers mean it does not, at the cost of running them. Where the data genuinely cannot leave, that decides the architecture.
Far less than people expect for format and tone — often a few hundred good examples. The work is in the quality of those examples, not the quantity, and that is where the time goes.
— other comparisons
Tell us the constraint.
Budget, timeline, or who maintains it after launch — whichever is hardest to change. We will tell you which option fits, including when it is not us.