InferenzLLM engineering
RAG or fine-tuning? How to choose
When a team wants a language model to know about their business, two techniques come up first: retrieval-augmented generation (RAG) and fine-tuning. They solve different problems, and mixing them up is a common and expensive mistake.
What each one does
RAG keeps the model unchanged. At question time it searches your documents or databases, puts the relevant passages into the prompt, and asks the model to answer from them. The model reads your knowledge each time.
Fine-tuning changes the model's weights by training on examples. It's good at teaching a style, a format or a behaviour, and less reliable at teaching facts.
A simple rule
Use RAG to add knowledge. Use fine-tuning to change behaviour.
How to choose
| If you need... | Start with |
|---|---|
| Answers from private or frequently changing documents | RAG |
| Citations so users can verify answers | RAG |
| A consistent tone, structure or output format | Prompting, then fine-tuning |
| Lower cost or latency for a narrow, high-volume task | Fine-tuning a smaller model |
| Per-user access control over what can be seen | RAG, with permissions on retrieval |
| Both fresh knowledge and a specific behaviour | RAG and fine-tuning together |
Why RAG usually comes first
- Freshness: update a document and the answers change. A fine-tuned model has to be retrained.
- Traceability: you can show the passages an answer came from.
- Permissions: retrieval can respect who is allowed to see what.
- Lower risk: it's easier to test and roll back than changing model weights.
When fine-tuning earns its place
Fine-tuning shines when prompting can't reliably get the output you need, when the task is narrow and repeated at high volume, or when you want a smaller, cheaper model to match a bigger one on your specific task. It needs a good set of labelled examples and an evaluation set, so it's usually step two, not step one.
Whichever you choose, measure it
Build a fixed set of real questions with known good answers before you start. For RAG, check both retrieval (did we find the right passage?) and the answer (was it correct and supported?). Most RAG failures are retrieval failures, so improving chunking, search and ranking often helps more than switching models.
If you're unsure which path fits your case, this is exactly what we work out in a discovery sprint.