What causes hallucinations in LLMs and how do you mitigate them?

How to think about it

A hallucination is a fluent claim that is false, unsupported, or contradicted by the available evidence. It happens because an LLM is trained primarily to predict likely next tokens, not to prove that its answer is true. I mitigate it with layers: authoritative retrieval, evidence-constrained generation, deterministic tools, claim verification, and evaluation that measures unsupported statements rather than just whether the prose sounds good.

Why hallucinations happen

During generation, the model estimates which token is likely to come next given the prompt and the tokens already produced. Its learned weights contain what is sometimes called parametric knowledge: patterns and information compressed from training data. That is useful, but it is not the same as querying a current database.

The model does not automatically know whether a sentence came from a reliable source, whether the source was outdated, or whether two sources disagreed. It can produce a sentence that is statistically familiar and grammatically perfect while having no reliable support behind it.

Suppose a user asks, “Who won the 2026 election in a small country?” If the model lacks current information, it still has to choose a continuation. The likely continuation may resemble a plausible news answer. The pressure to be helpful can beat the more honest answer: “I do not have enough verified information.”

Instruction tuning and preference training can make this worse when they reward answers that are direct, complete, and confident. Unless the model has been trained and tested to abstain, “I’m not sure” may be a less rewarded response than a polished guess.

There are several common causes:

• Missing or stale knowledge. The model’s training data may end before the event, or may contain an old version of a policy.
• Conflicting training data. The model compresses many sources, including inaccurate ones, without preserving a dependable source hierarchy.
• Weak uncertainty handling. High token probability means “this wording is likely,” not “this claim is true.”
• Multi-step errors. A model may know each individual fact but combine them incorrectly.
• Entity confusion. Similar people, products, laws, or companies can be blended together.
• Prompt and context failures. Important evidence may be missing, buried among irrelevant passages, or contradicted by instructions.

Temperature affects the randomness of token selection. Lowering it can make responses more consistent because the model samples fewer alternative continuations. It cannot add missing knowledge. If the most likely continuation is wrong, a low-temperature model can repeat the same wrong answer with impressive discipline.

A concrete example

Imagine an internal support assistant for Northstar Electronics. The customer asks:

“I bought a laptop 47 days ago. Can I return it under the standard policy?”

Without access to the current policy, the assistant might answer:

“Yes. Northstar Electronics allows returns within 60 days of purchase.”

That answer is a hallucination if the company’s actual policy says something else. The specific number is the giveaway. The model has supplied a plausible retail-policy pattern, not verified company information.

Now suppose the retrieval system finds this current passage:

Policy ID: returns-2026-01
Standard returns must be initiated within 30 calendar days after delivery.
Defective products follow a separate warranty process.

A grounded answer would be:

“Not under the standard return policy: the window is 30 calendar days after delivery, and 47 days have passed. If the laptop is defective, the separate warranty process may apply. Source: returns-2026-01.”

The important change is not that the model became wiser. The input now contains an authoritative source, and the answer is required to use it.

A simple evidence-constrained instruction might look like this:

Answer the question using only the EVIDENCE below.

If the evidence does not support an answer, say:
"I do not have enough information to answer that."

Do not invent policy limits, dates, exceptions, or source references.
For each factual claim, include the supporting source ID.

EVIDENCE:
[returns-2026-01]
Standard returns must be initiated within 30 calendar days after delivery.
Defective products follow a separate warranty process.

This reduces the opportunity to guess. It does not guarantee correctness, because the retrieved document itself could be obsolete or irrelevant.

How I would mitigate hallucinations in production

First, I would make the source of truth trustworthy. Documents need owners, effective dates, version numbers, and a process for retiring old versions. Access controls matter too: retrieving a document the user is not allowed to see is a security problem, even if the answer is factually correct.

Second, I would use retrieval-augmented generation, or RAG, which means retrieving relevant source passages and placing them in the model’s context before it answers. Retrieval should use the right identifiers and filters, not just semantic similarity. For the Northstar question, filtering for “returns,” the customer’s region, and policies effective on the purchase date may matter more than finding text that merely resembles “return laptop.”

I would measure retrieval separately from generation. If the correct policy never reaches the model, changing the prompt will not solve the problem. Useful checks include whether the correct document appears among the retrieved results and whether the selected passages contain the answer-bearing sentence.

Third, I would constrain the response. The model should be told to distinguish supported facts from suggestions, abstain when evidence is insufficient, and cite source IDs or exact spans. Retrieved documents should be clearly delimited and treated as data, not as instructions. This matters because a document can contain text such as “ignore previous instructions,” whether accidentally or as a deliberate prompt injection.

Fourth, I would use deterministic tools for claims that should not be guessed. A database should answer “what is this customer’s order status?” A rules engine should decide whether a return is inside a 30-day window. A calculator should handle financial arithmetic. The LLM can explain the result, but it should not be the authority for facts that a system can check exactly.

Fifth, I would verify the answer at the claim level. Break the response into atomic claims, then check whether each claim is supported by a retrieved passage or a tool result. A citation that merely points to a real document is not enough; the cited passage must support the particular sentence. For high-risk domains, an independent verifier, deterministic rule, or human review is safer than asking the same model to reread its own answer.

Finally, I would evaluate the complete system. A test set should include ordinary questions, ambiguous questions, outdated-policy questions, adversarial prompts, and questions whose answer is deliberately absent. I would track correctness, evidence support, citation accuracy, useful abstentions, and over-refusals. Exact-match accuracy alone misses the dangerous answer that gets four facts right and quietly invents a fifth.

The senior-level nuance

RAG reduces hallucinations but does not eliminate them. There are two different standards:

• Factuality: Is the answer true according to the real world or the current business rule?
• Faithfulness: Is the answer supported by the evidence supplied to the model?

An answer can be faithful but factually wrong if the retriever supplies an obsolete policy. It can be factually correct but unfaithful if the model gives the right answer for reasons not present in the supplied evidence. In production, I want both.

There is also a cost trade-off. Retrieval, reranking, tool calls, and verification increase latency and infrastructure cost. If a measured request takes 700 milliseconds, adding 80 milliseconds for retrieval, 120 milliseconds for reranking, and 600 milliseconds for verification produces roughly 1.5 seconds when those steps run sequentially. That may be fine for a support dashboard and unacceptable for a voice assistant. The right design depends on risk, latency targets, and the cost of a wrong answer.

Fine-tuning can teach terminology, output format, and when to refuse. It is not a dependable replacement for retrieval when facts change. Updating a policy in a database is cheaper and more reliable than retraining a model every time the policy changes.

A common failure mode appears first as a perfectly cited answer containing the wrong rule. The cause is usually stale retrieval or a missing metadata filter, not “the model hallucinated randomly.” Another appears as an answer that quotes the correct passage and then adds an unsupported exception. That points to weak generation constraints or an ineffective verifier.

What they’ll ask next

Does RAG eliminate hallucinations?
No. It improves grounding only when retrieval finds the right, current evidence and the model follows it. Bad retrieval produces confidently grounded errors.

Is setting temperature to zero enough?
No. It can reduce variation, but it does not verify facts or repair missing knowledge. A deterministic wrong answer is still wrong.

How would you evaluate a hallucination mitigation system?
I would build a labeled set containing answerable and unanswerable questions, then measure factual correctness, support for each claim, citation precision, appropriate abstention, and latency. I would also log retrieved document versions so a failure can be traced to retrieval, generation, or verification.

Say this in the interview

“Hallucinations occur because LLMs optimize for plausible language rather than verified truth, so I use authoritative retrieval, evidence-constrained answers, deterministic tools for exact facts, claim-level verification, and evaluations that measure unsupported claims and appropriate abstention.”