A product team wants the model to answer questions about frequently changing internal policies while also adopting a consistent response style. Which parts would you solve with RAG, which with fine-tuning, and what evidence would change your decision?

How to think about it

I would use RAG for the changing internal policies, and use fine-tuning only if evaluation shows that prompts and examples cannot produce a reliable response style. In practice, I would start with RAG plus a strong system prompt and representative examples, then fine-tune the style if the remaining inconsistency is frequent, measurable, and worth the operational cost.

Why this is the right split

RAG, or retrieval-augmented generation, gives the model relevant documents at question time. A typical pipeline takes the user’s question, searches an indexed policy collection, places the best passages into the model’s context, and asks the model to answer from those passages. The model’s weights do not change when Human Resources publishes a new policy.

That matters for a policy assistant. If the parental-leave policy changes on Friday, the production fix should be to publish and index the new document, not to collect training data, run a training job, validate a new model, and redeploy it. RAG can also attach citations or document dates, which gives an employee and an auditor a way to inspect the answer.

Fine-tuning is different. It updates the model’s parameters using examples so that the model is more likely to produce a particular kind of output. The examples might teach it to write in a calm, concise tone, ask for a country before answering a regional policy question, lead with the decision, and finish with a source citation.

The key distinction is simple:

Need	Better first tool	Reason
“What is the current travel-reimbursement limit?”	RAG	The answer lives in a document that can change
“Answer in three short paragraphs, name the exception, and cite the source”	Prompting, then possibly fine-tuning	This is behavior and formatting, not changing knowledge
“Use the policy effective on the employee’s hire date”	RAG with metadata and filtering	The system must select the right version, not merely recall a fact
“Do not invent an answer when the policy is silent”	Prompting, retrieval design, and evaluation; possibly fine-tuning	Abstention is behavior, but retrieval quality still controls the evidence

Fine-tuning is not a dependable policy database. A model may memorize training examples, but that memory is difficult to inspect, update, or remove. It may also blend old and new rules when the documents contain similar language. The weights are a bad filing cabinet.

A concrete example: the 3 a.m. relocation question

Suppose a company has 4,000 employees in the United States, United Kingdom, and India. Its relocation policy has changed three times in six months. The current US policy says that a qualifying employee can claim up to 8,000 dollars for moving expenses, but only if the move is approved before the start date. The UK policy uses pounds and has a different approval rule.

An employee asks:

“I start in Chicago next month. Can I claim 10,000 dollars for my move, and do I need approval first?”

A sensible RAG system would:

1. identify the employee’s region as the United States;
2. retrieve the current relocation policy and its effective date;
3. filter out archived versions;
4. supply the relevant passages to the model;
5. answer that the documented limit is 8,000 dollars and approval is required before the start date;
6. cite the policy version and perhaps say that the 10,000-dollar amount is not supported by the retrieved document.

The model is not expected to remember the 8,000-dollar number. It is expected to read the current source.

The response style could be specified with a system instruction:

State the decision first. Give the limit and condition in plain language. If the policy does not answer part of the question, say so. Cite the policy title and effective date.

Add 20 to 50 carefully written examples covering direct questions, exceptions, missing information, and unsupported claims. That may be enough. Fine-tuning becomes a candidate if the assistant still produces long, inconsistent answers after the retrieval and prompt design are sound.

For example, an evaluation set of 200 realistic employee questions might show that the assistant gives the required structure on 176 questions, or 88 percent, while the product target is 97 percent. If a fine-tuned model raises that result to 97 percent on held-out questions without increasing unsupported claims, it has a defensible role. The numbers are not a universal threshold; they are the kind of evidence I would want before paying the complexity bill.

The senior-level nuance: RAG can still be wrong

“Use RAG for freshness” is directionally right, but RAG does not make information current by magic. If the ingestion job misses the Friday update, the vector index still serves Thursday’s policy with impressive confidence. If retrieval returns a similarly worded archived document, the model may answer fluently from the wrong version.

I would test the whole chain, not just the language model:

• Freshness: How long after publication does a document become searchable?
• Retrieval recall: Does the correct policy passage appear in the top results?
• Version control: Are effective dates, regions, departments, and archived documents handled correctly?
• Answer grounding: Does the response stay within the retrieved evidence?
• Abstention: Does it decline or escalate when the documents do not answer the question?
• Citation correctness: Does each citation actually support the sentence attached to it?
• Style consistency: Does it follow the desired structure across ordinary questions and edge cases?

A useful failure symptom is a confident answer quoting an old dollar limit after a policy update. That points first to document ingestion, metadata filtering, or retrieval—not automatically to fine-tuning. Another symptom is that the correct passage appears in the retrieved context but the model ignores it and invents an exception. That could justify a better prompt, a stronger model, output constraints, or fine-tuning after controlled testing.

There is also a subtle style trap. If “consistent style” really means a fixed output shape, such as three fields named decision, reason, and source, structured generation or application-side validation may be better than fine-tuning. If it means “sound empathetic when denying a request,” examples and fine-tuning may help. The right tool depends on whether the behavior is a hard format rule or a learned preference.

What evidence would change my decision?

I would change the design based on measurements, not on whether fine-tuning or RAG is fashionable.

I would lean more heavily on prompting and not fine-tune if:

• style compliance is already above the product target;
• the failures are mostly caused by missing or incorrect retrieved context;
• the style requirements are likely to change next month;
• the team has too few high-quality examples to represent the real question distribution;
• latency, training cost, or deployment complexity matters more than a modest consistency improvement.

I would consider fine-tuning if:

• the desired style is stable for the foreseeable future;
• there are enough reviewed examples, including difficult and adversarial cases;
• held-out evaluation shows a repeatable style gap;
• the fine-tuned model improves that gap without worsening factual accuracy, refusal behavior, or citation quality;
• the gain remains after testing new policy topics, not only questions that resemble the training examples.

I would reconsider RAG if the policy collection is tiny, stable, and already encoded in a controlled application workflow. For a single unchanging rule, a normal database lookup or business-rule service may be safer than asking a language model to interpret a document. Conversely, if answers require comparing several current policies, RAG becomes more valuable because it can assemble the relevant evidence at request time.

The production design I would propose is therefore hybrid but deliberately asymmetric: RAG owns current facts; the application owns access control, document versioning, and deterministic rules; the prompt owns the initial style; fine-tuning is an evaluated optimization for residual behavior problems.

What they’ll ask next

“Why not fine-tune the latest policy into the model?”
Because the policy will become stale and the update path is slow and expensive. Fine-tuning also makes it harder to prove which source supported an answer. I would fine-tune stable behavior, not rapidly changing facts.

“Does RAG guarantee factual answers?”
No. It improves access to relevant evidence, but bad ingestion, weak retrieval, wrong version selection, or careless generation can still produce a wrong answer. I would measure retrieval and grounded-answer quality separately.

“How would you evaluate the system?”
I would build a held-out set of real policy questions with expected answers, source documents, effective dates, and approved abstentions. I would measure retrieval recall, citation support, factual accuracy, style compliance, and behavior after policy updates. I would also test permissions so an employee cannot retrieve a policy they are not allowed to see.

One line to say in the room

“RAG should own changing policy facts and their evidence; fine-tuning should be reserved for stable response behavior that prompts cannot reliably achieve, and I’d let freshness, grounding, style, and cost evaluations decide the boundary.”