Raphael Ting"Should we fine-tune or use RAG?" is the first architecture question on almost every AI project. The honest answer is that they solve different problems — and the best systems often use both. Here is how we decide at Code Language Hub.
The one-sentence distinction
RAG changes what the model knows. Fine-tuning changes how the model behaves. Keep that sentence in mind and most decisions become obvious.
What RAG does
Retrieval-Augmented Generation leaves the model untouched. At query time, you search a knowledge base (usually a vector database) for the most relevant chunks and paste them into the prompt. The model answers using that supplied context — so knowledge is external, fresh, and citable.
What fine-tuning does
Fine-tuning adjusts the model's own weights on your examples. It bakes in a behaviour — a tone, a format, a task pattern, a domain vocabulary — so you no longer need to describe it in every prompt.
Side-by-side
| Dimension | RAG | Fine-Tuning |
|---|---|---|
| Best for | Facts & knowledge that change | Style, format, task behaviour |
| Freshness | Update the database anytime | Requires retraining |
| Citations | Natural — you know the source | Not inherent |
| Upfront cost | Lower (build a pipeline) | Higher (prepare data + train) |
| Per-query cost | Higher (bigger prompts) | Lower (shorter prompts) |
| Risk | Retrieval misses = wrong context | Overfitting / stale knowledge |
A decision framework
Ask these in order:
- Does the answer depend on knowledge that changes? (prices, policies, inventory, docs) → RAG.
- Do you need answers traceable to a source? → RAG.
- Do you need a consistent tone, format or narrow task? → Fine-tuning.
- Are prompts getting long and expensive from repeated instructions? → Fine-tuning to compress that behaviour.
- Both of the above? → Do both.
The common production pattern: fine-tune a small model so it always replies in your brand voice and format, then attach RAG so every answer is grounded in your current documents. Behaviour from fine-tuning, facts from retrieval.
Worked example: a support agent
Say you want an AI support agent for an online school.
- RAG supplies the current course list, schedules, and fees from a database — so answers are always up to date and cite the source.
- Fine-tuning teaches the agent to always reply in two short sentences, in a friendly bilingual tone, and to escalate refunds to a human.
Neither alone is enough: RAG without fine-tuning rambles off-format; fine-tuning without RAG confidently quotes last year's fees.
Before you reach for either
Start with good prompting and a strong base model. A surprising amount of "we need fine-tuning" turns out to be "we needed a clearer prompt and a couple of examples". Reach for RAG when knowledge must be external and fresh; reach for fine-tuning when behaviour must be consistent and cheap to run at scale.
Key takeaways
- RAG = knowledge; fine-tuning = behaviour.
- Choose RAG for fresh, citable facts; fine-tuning for style, format and narrow tasks.
- Most robust production systems combine both.
- Try better prompting first — it's the cheapest lever.
New to fine-tuning specifically? Read our step-by-step guide: How to Fine-Tune a Small Language Model.