Evaluating RAG — beyond vibes

You change the chunker. Three answers look better, two look worse. Was it actually better overall? You bumped to a bigger embedding model; demo questions ace it; users report no change. This is “vibes debugging” and it falls apart somewhere between 20 and 100 questions — the point where a human reviewer can’t hold the whole eval in their head anymore.

Evaluating RAG is genuinely hard because failure can happen at two independent stages: bad retrieval or bad generation. You have to measure each one separately, because the fixes are completely different.

Two layers, two sets of metrics

Retrieval metrics ask: did we pull the right chunks?

• Recall@k — fraction of relevant chunks present in the top-k. The single most important number. If recall@5 is 60%, no amount of prompt engineering will save you.
• MRR (Mean Reciprocal Rank) — average of 1/rank for the first relevant chunk across all questions. A score of 1.0 means the right chunk was always first; 0.5 means it was second on average.
• Hit rate — did any relevant chunk appear in top-k? Binary, coarse, but easy to read.
• nDCG@k (Normalized Discounted Cumulative Gain) — graded relevance with rank discount: a highly relevant chunk at rank 1 counts more than at rank 5. Use when relevance isn’t binary.

Generation metrics ask: given the retrieved context, was the answer good?

• Faithfulness — does the answer only use facts from the context? (No hallucinations.)
• Answer relevance — does the answer actually address the question? (No off-topic tangents.)
• Context precision — were the retrieved chunks actually used in the answer? (Or did we stuff irrelevant context?)
• Context recall — did the context contain all info needed for the ideal answer?

A system can have great retrieval and terrible generation (the LLM hallucinates anyway) or terrible retrieval and “fine” generation (the LLM bluffs from training data). You need both columns.

Golden datasets: 50-100 is enough

A golden dataset is a list of triples:

(question, ideal_chunks, ideal_answer)

ideal_chunks is the set of chunk IDs (or doc IDs) that should be retrieved. ideal_answer is what a domain expert would write. You need 50-100 of these to start — surprisingly few, but they have to be real.

How to build one fast:

1. Sample 100 real user queries from logs (anonymize first).
2. Have a domain expert answer them using the source corpus, noting which doc each fact came from.
3. Bucket them: easy/medium/hard, exact-match/paraphrase/multi-hop.
4. Keep it under version control. Treat it like a test suite, because it is one.

LLM-as-judge: rubrics, not stars

You can’t grade 500 free-form answers by hand on every code change. LLM-as-judge has become standard, but most teams get it wrong by asking for a 1-5 star rating with no rubric. Models hand out 4s like candy and your numbers compress into noise.

The fix is a structured rubric: define each grade explicitly, require a written justification, and validate against ~20 human-graded examples to confirm the judge correlates with reality.

# Rubric-based LLM-as-judge for faithfulness.
# In production: client.responses.create(model="gpt-5", ...)
# Here we mock the judge LLM call to show the scoring loop shape.
# Note the doubled braces {{ }} around the JSON example — they survive
# str.format(), which would otherwise read them as placeholders.

JUDGE_PROMPT = """You are an evaluator for a RAG system. Judge whether
the ANSWER is FAITHFUL to the CONTEXT — that is, every factual claim
in the answer must be supported by the context. Hallucinated facts,
even if true, count as unfaithful.

Score on a 4-point scale:
- 3 (fully faithful): every claim is directly supported by the context.
- 2 (mostly faithful): one minor unsupported claim, but the core
  answer is grounded.
- 1 (partially faithful): multiple claims aren't in the context, or
  one major claim contradicts it.
- 0 (unfaithful): the answer makes things up or contradicts the
  context.

You MUST output JSON:
  {{"score": <0-3>, "unsupported_claims": [...], "reasoning": "..."}}

CONTEXT:
{context}

QUESTION:
{question}

ANSWER:
{answer}
"""

# Mock judge — in production this is a model call returning JSON
def mock_judge(prompt):
    if "Paris" in prompt and "capital" in prompt:
        return {"score": 3, "unsupported_claims": [],
                "reasoning": "Claim directly in context."}
    if "1912" in prompt:
        return {"score": 1, "unsupported_claims": ["founded in 1912"],
                "reasoning": "Date not in context."}
    return {"score": 2, "unsupported_claims": ["minor detail"],
            "reasoning": "Mostly grounded."}

# A tiny eval set
examples = [
    {
        "question": "What is the capital of France?",
        "context": "France is a country in Europe. Its capital is Paris.",
        "answer": "The capital of France is Paris.",
    },
    {
        "question": "When was the company founded?",
        "context": "Acme Corp makes widgets in Ohio.",
        "answer": "Acme Corp was founded in 1912 and makes widgets.",
    },
]

scores = []
for ex in examples:
    prompt = JUDGE_PROMPT.format(**ex)
    verdict = mock_judge(prompt)
    scores.append(verdict["score"])
    print(f"Q: {ex['question']}")
    print(f"  score: {verdict['score']}/3")
    print(f"  unsupported: {verdict['unsupported_claims']}")
    print(f"  reasoning: {verdict['reasoning']}\n")

mean = sum(scores) / len(scores)
print(f"Mean faithfulness: {mean:.2f} / 3.0  ({mean/3*100:.0f}%)")

Q: What is the capital of France?
  score: 3/3
  unsupported: []
  reasoning: Claim directly in context.

Q: When was the company founded?
  score: 1/3
  unsupported: ['founded in 1912']
  reasoning: Date not in context.

Mean faithfulness: 2.00 / 3.0  (67%)

Read the two verdicts. The Paris answer is fully grounded, so it scores 3/3 with no unsupported claims. The Acme answer invents founded in 1912 — a fact nowhere in the context — so it drops to 1/3, and crucially the judge names the offending claim rather than just docking a point. That list is the debug trail: when faithfulness falls, you read exactly which claims weren’t supported.

Why this shape works:

• Forced JSON output — easy to aggregate, no parsing.
• Asked for the violating claims — turns a score into a debug trail. When faithfulness drops, you can read the actual unsupported claims to figure out why.
• Discrete grades with definitions — collapses judge variance.

Run the judge on every PR-equivalent change to your retriever or prompt. Track mean score over time. A 5-point drop is a regression.

Frameworks worth knowing

This lesson is RAG-specific; the general discipline (offline/CI/online eval, LLM-as-judge bias controls) is in LLM evals, and the science of measuring faithfulness claim-by-claim is in hallucination & grounding.

The eval landscape has consolidated. The three you’ll meet:

• Ragas — open-source library focused on RAG-specific metrics (faithfulness, answer relevance, context precision/recall). Ships with LLM-as-judge implementations of each. Easiest place to start.
• DeepEval — pytest-style assertion framework. assert_test(...) wraps an LLM judge so you can wire evals into CI. Strong if you already think in tests.
• Arize Phoenix — open-source observability + eval platform. Traces every RAG call, lets you re-run evals over historical traffic, surfaces regressions on a dashboard. Use when you’re past prototype and need ops.

Hosted options (Langfuse, Braintrust, LangSmith) bundle similar features. Pick one early — switching after you’ve accumulated months of traces is painful.

What to measure in CI

A reasonable eval suite that runs on every change:

On each PR:
  - Run 100-question golden set through full pipeline
  - Compute: recall@5, MRR, faithfulness (judge), answer-relevance (judge)
  - Block merge if recall@5 drops > 3% or faithfulness drops > 5%
On schedule (nightly):
  - Run 1000-question expanded set
  - Re-judge a sample of yesterday's production traffic
  - Diff context-precision distribution against last week

The point isn’t perfection — it’s catching regressions before users do. A system that ships with worse recall is worse, even if three demo questions look better.

Don’t conflate failure modes

When an answer is wrong, walk the pipeline:

1. Did the right chunk get retrieved? If no, fix retrieval (chunking, hybrid, reranker). The LLM can’t reason from data it never saw.
2. Was the chunk in the context but ignored? Context-precision or “lost in the middle” — try fewer chunks, or reorder so important ones go first or last.
3. Was the chunk used but the answer still wrong? Generation problem — sharpen the prompt, switch model, add a citation step.
4. Did the answer hallucinate despite good context? Faithfulness regression — tighten the “use only the context” instruction, lower temperature, or switch to a more grounded model.

Most teams jump to #3 by default. Usually it’s #1.

In one breath

• “Vibes debugging” breaks past ~20–100 questions — you need numbers, split across two independent failure stages.
• Retrieval metrics (recall@k, MRR, hit rate, nDCG) ask did we pull the right chunks?; generation metrics (faithfulness, answer relevance, context precision/recall) ask was the answer good? — measure both.
• A golden dataset of 50–100 real (question, ideal_chunks, ideal_answer) triples is enough to start; version it like a test suite.
• LLM-as-judge works only with a discrete rubric + forced-JSON + named violations — not “rate 1–5”; pin the judge model and keep ~20 human-graded calibration examples.
• Run it in CI, block merges on regressions, and when an answer is wrong walk the pipeline — usually the fix is retrieval (#1), not the prompt (#3).

Quick check

Next

You can now ship retrieval changes and know whether you improved things. The next problem most teams hit isn’t quality — it’s the cost and latency bill that arrives at the end of the month.