Compare RAG and fine-tuning. When would you use each?

How to think about it

Use RAG, or retrieval-augmented generation, when the system should fetch relevant source text for each request. Use fine-tuning, or additional training on examples that changes the model’s learned parameters, when the model should behave differently across requests. RAG changes what the model can see; fine-tuning changes how it tends to respond.

The distinction the interviewer is testing

A language model stores patterns in its parameters, the numerical values learned during training. Those parameters are useful for language, reasoning patterns, and broad background knowledge. They are a poor place for a frequently changing company policy.

RAG gives the model evidence at request time. A retriever searches a document collection, selects relevant passages, and places them in the prompt sent to the language model. The model then generates an answer using that context. Change the source document, update the index, and the next request can use the new version without retraining the model.

Fine-tuning works earlier in the lifecycle. You provide examples such as a user request paired with an ideal answer, classification, or tool call. Training adjusts the model’s parameters, or an adapter attached to them, so that the desired behavior becomes more likely. That is useful for tone, structure, classification rules, domain vocabulary, or consistent tool-use patterns.

The cleanest mental test is this:

• If the request is “What is the current answer?” think RAG.
• If the request is “How should the model answer this kind of request?” think fine-tuning.

Fine-tuning can encode facts, especially stable and repeated facts, but that does not make it a dependable database. The model may recall the fact inconsistently, blend it with a similar fact, or retain an old version after the source changes. RAG can supply current facts, but it does not automatically teach the model a new skill. The model still needs instructions or training that explain how to use the retrieved material.

A concrete example: Northstar’s benefits assistant

Imagine Northstar has 50,000 internal benefits documents. Each averages 500 tokens, so the collection contains roughly 25 million tokens. An employee asks:

“I moved from Texas to New York. Can I take 12 weeks of parental leave, and what paperwork is due?”

The answer may depend on location, employment type, hire date, and the effective date of the policy. It may change next month. It may also be restricted to employees who are allowed to see particular documents.

A RAG system would typically:

1. Split documents into searchable passages while preserving titles, policy versions, effective dates, and access metadata.
2. Create embeddings, which are numerical representations that help search by meaning rather than only exact words.
3. Filter results using the employee’s permissions and relevant metadata such as region.
4. Retrieve and possibly rerank a small set of passages. For example, six passages at 350 tokens each add about 2,100 tokens to the request instead of sending all 25 million.
5. Ask the model to answer only from those passages and include their document names and effective dates.

If the current policy says, illustratively, that an eligible New York employee receives 12 weeks and must submit a form 30 days before leave, the assistant can quote that passage and link to the policy version. When Northstar replaces the policy, the source and index can be updated without another model-training run.

Now suppose Northstar also has 12,000 reviewed support tickets. The tickets show how agents classify requests, when they escalate them, and what response format the company wants. A fine-tuned model could learn to return a stable structure such as:

{
  "category": "parental_leave",
  "escalate": true,
  "response_style": "concise_and_formal"
}

That training can make the model much more consistent about classification, escalation, and formatting. It should still use RAG to obtain the actual current leave policy. The strongest design is therefore often a fine-tuned response behavior wrapped around retrieved evidence.

Where each approach fits

Concern	RAG	Fine-tuning
Changing facts	Strong fit because the index can be refreshed	Weak fit because new facts usually require more training
Citations	Natural if source metadata is preserved	Difficult because the answer comes from learned parameters
Private or permissioned knowledge	Good if access checks happen before retrieval	Risky because knowledge is not naturally isolated per user
Tone and output format	Possible through prompting, but may vary	Strong fit when examples consistently show the desired behavior
New classification or extraction task	Often enough for simple cases	Useful when the task is repeated and prompts are not consistent enough
Runtime cost	Adds search, reranking, and context tokens	Adds training and model-versioning cost; may shorten prompts
Main failure	Relevant evidence is not retrieved	The model learns stale, narrow, or conflicting behavior

The distinction is not about which technique is more advanced. It is about where the uncertainty lives. RAG addresses uncertainty in the information. Fine-tuning addresses uncertainty in the behavior.

The senior-level nuance

RAG is not “put documents in a vector database and ask a model a question.” Retrieval quality is a separate system to evaluate. A semantically similar passage can still be the wrong policy version. Exact terms, dates, product identifiers, and access permissions often call for keyword search or metadata filters alongside vector search. Many production systems use hybrid retrieval and a reranker rather than trusting one similarity score.

RAG also does not guarantee truthful answers or valid citations. The application must preserve source identity, instruct the model not to invent unsupported claims, and check whether the cited passage actually supports the sentence. A citation that merely exists is not a citation that proves the answer.

Fine-tuning has its own trap. A model can become excellent at producing the requested JSON while becoming no better at factual accuracy. It can also overfit to the wording in its examples: the offline test looks impressive, then a customer uses a synonym and the model misses the intent. There is no universal rule that a particular number of examples makes fine-tuning worthwhile. Example quality, coverage, label consistency, base-model capability, and evaluation design matter more than a magic count.

I would not fine-tune a model merely because the company has a folder of documents. Raw documents are knowledge, not demonstrations of desired behavior. I would start with RAG when the documents change, require citations, or contain user-specific permissions.

I would not add RAG to a stable, closed task that has no external facts. If the job is to classify 20,000 support messages into six fixed categories, retrieving a document for every message may add latency and another failure point. A prompt, a structured output constraint, or fine-tuning may be the cleaner solution.

Conversely, I would not use fine-tuning alone for a policy assistant whose answer must reflect yesterday’s update. The training set becomes a second, hidden copy of the policy. Someone will update the official document, forget to retrain the model, and discover the problem during the 3 a.m. incident rather than during a design review.

Permissions are especially important. RAG can apply an employee’s access control before putting passages into the prompt. Fine-tuning makes selective revocation much harder: if sensitive material influences the learned parameters, deleting one document from storage does not reliably remove its influence from the model.

Failure modes I would look for first

A RAG system’s first visible symptom is often a confident answer with a citation that sounds related but does not support the claim. The underlying cause may be that the relevant passage was not among the retrieved results, the chunk was split away from its exception clause, or an older document outranked the current one. I would inspect retrieval results separately from generation and measure whether a known relevant passage appears in the first few results.

A fine-tuned system’s first visible symptom is often excellent formatting paired with stale or invented facts. For example, the assistant returns valid JSON and a polished answer, but still quotes Northstar’s 2025 leave policy after the 2026 policy took effect. That is a behavior success and a knowledge failure. Moving the changing facts into RAG addresses the cause more directly than adding more examples of old answers.

What they’ll ask next

“Can fine-tuning teach the model our private documents?”

Technically, it can make some information more likely to appear, but I would not use it as the source of truth for changing private documents. Updates require another training cycle, citations are difficult, and selective deletion or per-user permissions are awkward. I would retrieve the documents at runtime and fine-tune only the response behavior if needed.

“How would you improve a RAG system that gives wrong answers?”

I would separate retrieval evaluation from answer evaluation. First, build a test set where each question has one or more known relevant passages, then measure whether those passages appear in the first few results. Next, inspect chunk boundaries, metadata filters, document versions, query rewriting, and reranking. Finally, evaluate answer correctness, groundedness, citation support, and appropriate abstention. If the right passage never arrives, changing the language model is treating the symptom.

“Would you ever combine RAG and fine-tuning?”

Yes. I might fine-tune the model to classify the request, follow a company response format, escalate safely, or call a tool consistently. At runtime, RAG would provide the current facts and citations. The evaluation should compare a baseline, RAG alone, fine-tuning alone, and the hybrid system, because the extra machinery must earn its latency and maintenance cost.

Say this in the interview: “RAG is for changing, private, or citable knowledge at request time; fine-tuning is for stable behavior, style, or format, and I would often retrieve the facts while fine-tuning how the model uses and presents them.”