How would you evaluate a RAG system so that you can tell whether a bad answer came from retrieval, context assembly, or generation? Which offline and online metrics would you trust, and how would you use an LLM judge without letting it hide regressions?

How to think about it

I would evaluate retrieval-augmented generation, or RAG, as three separately logged contracts: did retrieval fetch the evidence, did context assembly put the right evidence into the actual prompt, and did generation answer only from it? I would trust labeled retrieval recall and task-level grounded correctness offline, sampled human-verified outcomes online, and use an LLM judge as a calibrated assistant rather than the release gate.

Why the separation matters

A RAG answer is a pipeline, not a single model call.

Retrieval searches the document collection and selects passages. Context assembly reranks, deduplicates, formats, and fits those passages into the model’s prompt. Generation then writes the answer. A failure at any stage can produce the same user-visible symptom: “That answer is wrong.”

The remedies are different.

If the required policy paragraph never appears in the retrieved set, changing the prompt will not help. Improve indexing, chunking, query rewriting, filters, or ranking.

If the paragraph was retrieved but was removed when the prompt hit its token budget, retrieval is innocent. Fix truncation, packing, ordering, or context compression.

If the paragraph is plainly present in the final prompt and the model still invents a rule, that is a generation or instruction-following problem. Change the model, decoding constraints, prompt, or refusal policy.

That means every evaluation record should preserve the query, retrieved chunk IDs and scores, the exact assembled context, the final prompt, the answer, and citations. Without that trace, root-cause analysis becomes archaeology.

A concrete diagnostic

Imagine a customer-support assistant answering refund questions at 3 a.m. The governing article says:

• returns are allowed within 30 days;
• refunds go to the original payment method;
• final-sale items are excluded.

Build a test set of 100 answerable questions with human-labeled evidence spans, plus 20 unanswerable questions such as “Can I return a used gift card?” The evidence label should identify the passage and, where practical, the required facts inside it. A document ID alone is too coarse: a 2,000-word article can contain both the rule and its exception.

Now run the same generator under controlled conditions:

1. Retrieval recall at five: the required evidence appears somewhere in the top five results for 92 of 100 questions.
2. Final-context evidence coverage: after reranking, deduplication, and a 1,200-token context budget, all required evidence survives for 87 questions.
3. Oracle-context generation: give the model the human-selected evidence directly. It produces a correct answer for 91 questions.
4. Production-context generation: give it the context assembled by the real pipeline. It gets 78 correct.

Those figures are not one additive score. They are counterfactual probes. The gap from 92 to 87 points to assembly losses. The gap between oracle-context and production-context correctness points to context quality plus generation sensitivity. On an individual failed question, the trace gives the sharper diagnosis:

Observation	Likely fault
Required passage is absent from top five	Retrieval
Passage is retrieved but absent from final prompt	Context assembly
Passage is present, but the answer contradicts it	Generation
Answer cites a passage but adds an unsupported exception	Generation or citation attribution
Question is unanswerable, but the system confidently answers	Abstention policy and generation

For multi-fact questions, “some relevant text was retrieved” is not enough. The answer needs every material fact. A context containing “30 days” but omitting “final-sale items are excluded” has poor evidence coverage even though a semantic-similarity metric may celebrate it.

Offline metrics I would trust

Retrieval recall at k is the first useful measure. Recall at five asks whether at least one relevant passage appears in the top five; for multi-hop questions, require all labeled evidence pieces. Recall matters more than retrieval precision when missing one safety-critical clause makes the answer impossible. Still, very high recall achieved by stuffing 50 noisy chunks into the prompt can harm generation, so I would also inspect ranking metrics such as reciprocal rank or nDCG when ordering matters.

Final-context evidence coverage measures what the model actually received, not what the retriever intended to send. It should catch truncation, duplicate chunks, bad metadata filters, lost citations, and contradictory passages. I would calculate it from labeled evidence spans and also check whether each cited claim maps to the supporting span.

Answer correctness is different from groundedness. Correctness asks whether the answer reaches the right conclusion against a reference answer, a verified fact set, or a domain-specific checker. Groundedness asks whether the answer’s claims are supported by the supplied context. A response can be grounded in a stale document and still be wrong. It can also reach the right answer while inventing an unsupported reason.

I would score these separately:

• claim correctness;
• groundedness of each material claim;
• completeness of required facts;
• citation precision, meaning cited text really supports the claim;
• citation coverage, meaning important claims have support;
• correct abstention on unanswerable questions.

For the 20 unanswerable refund questions, measure both false answers and false refusals. “I do not have enough evidence” is valuable only when evidence really is missing. A system that refuses every difficult question can look wonderfully safe while doing no useful work.

Finally, measure latency, token use, and retrieval failure rates. A quality improvement that adds 4 seconds and doubles context cost may be the wrong production choice, especially for a high-volume support queue.

Online metrics and the LLM judge

Online, I trust task success and audited correctness more than engagement proxies. Thumbs-up, click-through, and conversation length are useful signals, but they are biased: users who silently accept an answer may have missed the error, while users who ask a follow-up may simply want more detail.

I would run canaries or A/B tests and report:

• human-audited grounded correctness on a random sample;
• unsupported-answer and bad-abstention rates;
• user correction, escalation, and support-agent override rates;
• p50 and p95 latency;
• token cost and retrieval error rate;
• results by slice, including unanswerable questions, long documents, permissions, languages, and recently changed policies.

An LLM judge can scale the first pass, but it should receive a strict rubric and separate inputs: question, final context, answer, citations, and reference facts where available. Ask it to score correctness, groundedness, completeness, and refusal separately. Do not ask only whether the answer is “helpful.” That rewards fluency, confidence, and length—the three traits most capable of hiding a bad retrieval result.

I would calibrate the judge against human labels on a fixed sample, track agreement and disagreements, and keep the judge model and rubric versioned. For pairwise comparisons, swap answer order to detect position bias. For high-stakes claims, use deterministic fact checks or human review rather than trusting a judge’s prose explanation.

Most importantly, never let dimensions compensate for one another. A release should not pass because its overall judge score rose from 4.2 to 4.4 while the unanswerable-question failure rate doubled. Gate critical slices and hard measures such as evidence recall, required-fact checks, and unsupported-claim rate. Use confidence intervals or paired per-question comparisons, not just a pleasing average.

A common failure looks like this: the judge score improves, citations still point to a relevant refund article, but support agents report more wrong answers. The trace shows that a formatter truncated the final 400 tokens, removing the final-sale exception. The judge saw a fluent answer and a citation, but the rubric did not require checking every policy exception. That is precisely why the judge is a measurement tool, not an authority.

The senior-level trade-off

Maximizing retrieval recall is not automatically good. More chunks increase token cost and can introduce conflicting versions of a policy. Context position matters too; a model can technically receive the right paragraph and still overlook it among fifteen distractors.

I would therefore optimize the pipeline jointly, while diagnosing it separately: retrieve enough evidence, rerank for precision, assemble a compact context with explicit source boundaries, and test the generator with both clean evidence and realistic distractors. Also include stale-document and permission-boundary tests. Groundedness against the wrong document is still a production bug.

What they’ll ask next

“What if we have no labeled evidence set?”
Start with a small human-labeled slice. Use synthetic questions or existing citations to expand coverage, but treat them as weak labels until humans verify them. A hundred carefully labeled questions beat ten thousand guessed ones for root-cause diagnosis.

“Retrieval recall is high, but answers are still bad. What do you check?”
Compare production context with oracle context, then inspect the exact rendered prompt. Look for truncation, contradictory chunks, missing metadata, poor ordering, and unsupported claims. If oracle context also fails, the problem is primarily generation or the task specification.

“How do you stop the judge from hiding a regression?”
Keep a fixed and hidden regression set, compare paired examples, report slice metrics, calibrate against human labels, and require hard gates on retrieval recall, critical facts, abstention, and unsupported claims. The judge may explain a regression; it may not overrule it.

One line to use in the room

“I would log and score retrieval, final-context coverage, and generation as separate contracts, then use a calibrated LLM judge for scale while hard, slice-level checks and human audits prevent fluent answers from hiding regressions.”