Fine-tune vs RAG: the decision

At 3 p.m. on Monday, your support chatbot gives the right answer about SSO.

At 9 a.m. on Tuesday, the pricing team removes SSO from the Business plan. The chatbot keeps confidently promising it to customers because someone fine-tuned the model on last month’s documentation.

That is not a strange edge case. It is the natural result of putting changing facts into model weights.

The useful distinction is short:

RAG adds knowledge. Fine-tuning changes behavior.

RAG, or retrieval-augmented generation, finds relevant source material at request time and places it in the model’s context. Fine-tuning trains the model on examples so that its learned behavior changes. One supplies temporary information. The other changes response habits.

A support bot may need:

• current pricing;
• strict JSON;
• a particular tone; and
• reliable escalation to a human.

Those are four different engineering problems.

Start with the symptom, not the technology

Imagine Northstar Cloud, which sells three hosting plans and has 20,000 chunks of internal documentation: pricing pages, runbooks, API references, incident procedures, and migration notes.

The team says, “It doesn’t know Northstar.” That is not a diagnosis. Ask what happened in a failed answer.

• If it gives an outdated plan rule, the missing ingredient is a fact. Retrieve the current page with RAG.

• If it knows the answer but emits malformed JSON, use structured output or constrained decoding first. Fine-tuning may help select labels or values, but it cannot replace semantic evaluation.

• If it has the answer in the required format but misroutes a billing question versus a production incident, fine-tuning may improve that narrow skill.

• If it is accurate but too expensive at 100,000 requests per day, consider:

  • a smaller model;
  • routing;
  • caching; or
  • distillation.

  Fine-tuning is not automatically a speed button.

The first diagnostic question is:

Is the model missing information, or is it using available information badly?

What RAG actually does

For a question such as:

“Does the Business plan include SSO, and what is the current seat limit?”

a RAG system generally:

1. represents the question for search;
2. retrieves relevant document chunks;
3. places them in the prompt, often with source identifiers; and
4. generates an answer from that context.

The model’s parameters have not changed. It receives a temporary reading packet.

Assume Northstar has 20,000 indexed chunks and the retriever returns the best five. At 250 tokens per chunk, the prompt receives roughly 1,250 retrieved tokens, plus the question and instructions. The model need not remember every pricing rule in its weights; it needs to read the candidate passages.

When a pricing page changes, Northstar re-indexes that chunk. The next request can retrieve the new page without updating the model. That is the causal reason RAG suits changing knowledge: the source of truth is updated in the data layer, not baked into a training run.

RAG is not a truth machine. Retrieval can return the wrong chunk, bury the right one below the top five, or expose contradictory documents. The model can also ignore relevant evidence or follow malicious instructions in a passage.

RAG moves the failure boundary. Instead of asking the model to recall a fact from its parameters, you ask search to find evidence and the model to use it.

That is usually better for:

• facts that change;
• facts that need citations; or
• facts that must be removed quickly.

You still need:

• document ownership;
• freshness metadata;
• chunking;
• access controls;
• retrieval evaluation; and
• an answer policy for missing evidence.

What fine-tuning actually does

Fine-tuning continues training a pretrained model on curated examples. Each example effectively says, “Given this input and context, produce this kind of output.” Training adjusts parameters so outputs resembling those examples become more likely.

That naturally teaches regularities such as:

• classifying incidents into labels;
• calling tools with the right fields;
• writing in a consistent style;
• transforming messy tickets into a fixed format; and
• performing a narrow task that prompting handles unreliably.

Fine-tuning can memorize facts, but it is a poor reliable fact store.

Facts are distributed across parameters rather than stored as explicit records. There is no simple operation to replace “seat limit” in one row. A later run may partially overwrite an old association while related wording still produces the old answer.

The model also has no built-in freshness signal. It cannot know that a learned sentence is six months old unless the prompt or another system tells it. Memorization is difficult to verify: a model may answer a familiar training example perfectly but fail on a new product name, negation, or combination of rules. It has learned patterns that often predict the answer, not a queryable policy database.

Narrow additional training can also cause catastrophic forgetting, the loss or degradation of useful prior behavior. This is a risk when data is repetitive, narrow, or poorly mixed with the base model’s capabilities.

For Northstar, fine-tuning might teach:

“When the customer reports data loss, classify the ticket as a production incident, ask for the incident ID, and offer the human escalation tool.”

It should not be the primary mechanism for keeping this week’s pricing table current.

A worked decision with numbers

Here is a small, hypothetical Northstar evaluation. The numbers are illustrative; the point is to reason from measurements.

The team builds 500 test prompts:

• 250 ask about changing product or policy facts;
• 150 test the required response format; and
• 100 test incident routing and escalation.

With a strong system prompt and three examples:

The team adds retrieval over current documents, includes source IDs, and checks whether retrieved evidence contains the answer:

RAG improved the knowledge slice by 78 correct answers, from 145 to 223. It did not teach JSON or incident policy. That is what the mechanism predicts.

The team then finds that the top five chunks often contain an obsolete migration guide instead of the current pricing page. It adds version metadata, removes superseded pages from the default index, and evaluates retrieval separately.

Suppose retrieval recall rises from 84 percent to 96 percent. Recall is the fraction of test questions for which the relevant evidence appears in the retrieved set. The factual answer score might rise from 89.2 percent to 94 percent. Before fine-tuning for knowledge, measure whether knowledge is reaching the model.

The format slice remains 74 percent. A JSON Schema constrained-output mechanism is a better fit if the requirement is valid structure. Constraints prevent illegal syntax; they do not make the answer correct. See constrained decoding and structured outputs.

Incident routing is different. Northstar has 4,000 reviewed examples with stable labels. After prompt improvements, its held-out score is 78 percent. A LoRA adapter raises it to 93 percent while RAG remains in the system for live product facts.

That combination fits the failures:

• RAG supplies current plan rules and runbook details.
• The adapter makes routing more consistent.
• Constrained output enforces the machine-readable shape.
• Separate evaluations test each component.

The numbers do not prove that fine-tuning is generally better than RAG. They identify which component fails for this task.

The escalation order

Use the least invasive tool that fixes the measured failure.

Prompting and few-shot examples come first. They clarify roles, allowed actions, structure, and evidence policy at low cost.

Use RAG when the answer depends on external or changing information. It is especially valuable for citations, access permissions, and documents that must be updated or withdrawn independently of the model.

Fine-tune when a behavior gap survives a good prompt and adequate context. You need representative examples, a held-out set, and a clear target. “Make it smarter” is not a target; “choose one of six routing labels on new tickets” is.

Distillation trains a smaller model to imitate a stronger teacher on a chosen task. It is attractive when the task is narrow, traffic is high, and the quality target is understood. The teacher can still use RAG while generating training examples; distillation does not require putting current knowledge into the student’s weights.

This order is a default, not a rigid staircase. Privacy may rule out an external RAG service, an on-device model may need adapter tuning immediately, and a strict schema may call for constrained decoding before training. Escalate because evidence demands it.

LoRA and QLoRA: small changes to behavior

LoRA, or Low-Rank Adaptation, freezes the original model and adds trainable low-rank matrices to selected transformations. A full update to a matrix with 4,096 input and output dimensions contains:

4,096 × 4,096 = 16,777,216 values.

A rank-16 LoRA update uses:

4,096 × 16 + 16 × 4,096 = 131,072 values,

about 0.78 percent as many for that matrix. Real configurations attach adapters to several projections, so the overall percentage varies. The principle is unchanged: train a small directional adjustment while freezing the base.

At inference, the base and adapter work together. Several adapters can support different customers or tasks. Each adds:

• evaluation;
• storage;
• routing; and
• compatibility work.

QLoRA combines LoRA with a quantized frozen base. Quantization stores weights with fewer bits, reducing memory at the cost of approximation. Four bits per parameter means 7 billion raw weight values occupy roughly 3.5 billion bytes before scales, metadata, runtime buffers, and caches.

Actual memory depends on:

• sequence length;
• batch size;
• implementation; and
• optimizer state.

QLoRA makes training practical with less hardware; it does not create a compressed, live knowledge vault. Neither LoRA nor QLoRA creates an updating document index or guarantees valid JSON.

For the mechanics, see Fine-tuning: LoRA, QLoRA and PEFT.

Failure symptoms and the production trade-off

If a fine-tuned model invents plan details, especially on new questions, it probably learned document style and associations rather than a dependable lookup procedure.

Retrieve:

• current documents;
• source IDs; and
• questions not copied from training.

If RAG quotes a real but obsolete document, inspect retrieved chunks before changing the model.

Check:

• version;
• tenant permissions;
• source authority;
• chunk boundaries; and
• whether the correct passage appears at all.

If the correct passage appears, improve reranking, evidence layout, prompting, or the abstention policy. Retrieval and generation quality are separate measurements.

If a fine-tune scores 98 percent but fails on new customers, inspect:

• leakage;
• duplicates;
• temporal coverage;
• rare labels; and
• catastrophic forgetting.

Keep a held-out set with new wording and measure the original capabilities you still need. Lower training loss is not proof of production quality.

RAG keeps knowledge outside the model, making updates and citations easier. It adds:

• retrieval latency;
• context-token cost;
• indexing work;
• access-control risks; and
• possible prompt injection in retrieved text.

Treat retrieved content as data, not unquestioned instruction; see Prompt injection.

Fine-tuning can reduce prompt length and improve a narrow behavior, but it costs:

• curated data;
• training and evaluation time;
• deployment complexity; and
• rollback discipline.

It can encode bias or overconfidence, and updating a fact requires another controlled training cycle.

Measure these rather than assuming an adapter makes the system faster:

• retrieval time;
• time to first token;
• generation time;
• total latency; and
• cost.

See LLM cost and latency.

A useful final test is to write the failure as one sentence:

• “The answer needs a fact that lives in a changing source.” Use RAG.
• “The model sees the fact but performs the task incorrectly.” Try examples, then fine-tuning.
• “The output must obey a machine-checked shape.” Use constrained or structured output.
• “The result is good but serving costs too much.” Consider routing, caching, a smaller model, or distillation.
• “We have not measured the failure.” Do not train yet.

Quick check

Next

If you need to change a model’s behavior, see Fine-tuning: LoRA, QLoRA and PEFT. For reliable structure, start with constrained decoding or structured outputs. For a smaller model at high volume, see distillation. Use LLM evals to prove the change helped.