Design a RAG pipeline for questions that require joining facts from several documents, handling freshness, and producing citations. How would you decide between query decomposition, hybrid retrieval, reranking, iterative retrieval, and a retrieve-more-than-top-k strategy?

How to think about it

I would use a staged RAG pipeline, meaning retrieval-augmented generation, with hybrid retrieval for recall, reranking for precision, and explicit freshness and provenance metadata throughout. I would add decomposition when the question has separate subquestions, iterative retrieval when one hop reveals the next lookup, and a larger candidate pool whenever the relevant evidence is likely to be buried below the first few results.

Why this is the real problem

Imagine a support engineer asks:

Which customers are affected by the new retention policy, when does it take effect, and what exception should support apply?

The answer may not exist in one paragraph. One document defines the policy. Another maps product plans to retention settings. A third describes an exception in customer contracts. A fourth may be newer than all three.

This is a multi-hop question: the system must follow a chain of facts across documents. Ordinary top-five vector search often retrieves the policy but misses the plan mapping. The language model then fills the gap with a plausible sentence. That sentence may even have a citation attached, but the citation does not prove it. This is how polished misinformation gets into production.

The pipeline therefore has two separate jobs:

1. Find enough relevant evidence.
2. Assemble only claims that the evidence actually supports.

Those jobs need different tools. Retrieval favors recall, meaning finding the relevant material even if it also returns noise. Reranking and final context selection favor precision, meaning keeping the passages that best answer the question.

The production pattern

First, I would index chunks with more than text. Each chunk should carry a stable source identifier, document title, page or section, publication time, effective time, expiry time if applicable, version, authoritativeness, and access-control information.

Freshness is not simply “sort by upload date.” A policy uploaded yesterday may not take effect until next month. A document published last week may explicitly supersede one published yesterday. For time-sensitive questions, I would use the relevant date field, follow supersedes relationships where available, and retain old versions for questions such as “what was the policy in March?”

At query time, a lightweight planner extracts the entities, dates, constraints, and likely number of hops. It then chooses a retrieval plan.

Hybrid retrieval combines lexical search, such as BM25, with dense vector search. Lexical search is strong when the question contains an exact plan ID, error code, contract phrase, or version number. Dense search is strong when the user says “how long do we keep logs?” while the document says “data-retention duration.” Combining their ranked results, often with reciprocal rank fusion, protects against either search method missing the wording.

I would usually begin with hybrid retrieval because it is a broad improvement, not a special-case trick.

Query decomposition breaks one question into smaller searches while preserving shared entities and constraints. For the support question, the planner might create:

• What is the current retention limit and effective date?
• Which plans still use the old setting?
• What contract exception applies?
• What should support tell affected customers?

Each subquery runs through hybrid retrieval. The results are then joined using explicit entities such as plan IDs, policy versions, dates, and customer segments.

Reranking takes the combined candidate passages and scores each passage against the actual query, usually with a more expensive query-passage model than the initial embedding search. Embedding retrieval may find passages about retention. A reranker can prefer the passage that states the effective date and applies to the customer’s plan.

The important distinction is that reranking improves ordering, not recall. If the relevant document never enters the candidate pool, a reranker cannot rescue it.

Iterative retrieval is useful when the first search discovers the key needed for the second search. Suppose the first result says that the exception depends on a contract clause named “extended retention.” The system can use that phrase, plus the customer’s plan and region, for a second retrieval step. This is adaptive retrieval: the next query depends on evidence found in the previous hop.

I would cap the loop, usually at a small number of hops, and stop when every required subquestion has supporting evidence. Otherwise the system can wander through loosely related documents until it finds something that sounds agreeable. Search should not become an archaeological expedition with no permit.

Retrieve-more-than-top-k means separating the candidate count from the final context count. For example, retrieve 30 passages per subquery, deduplicate them, rerank 60 combined candidates, and place only the best 8 passages in the generation context. Calling this “top-k retrieval” without specifying which k causes endless confusion.

Here is the decision rule I would use:

Technique	Use it when	Main risk
Hybrid retrieval	Exact terms and paraphrases both matter	More candidates and score fusion to manage
Query decomposition	The question has identifiable subquestions or joins	Query drift and extra latency
Reranking	Initial search finds relevant material but orders it poorly	Compute cost; no recall recovery
Iterative retrieval	A first hop reveals an ID, name, clause, or missing bridge	Loops and error propagation
Larger candidate pool	Evidence is scattered or the first results are unstable	More reranking cost and possible context noise

These techniques are complementary. A common plan is decomposition, hybrid retrieval for each branch, union and deduplication, reranking, then one short iterative lookup if a bridge is still missing.

A concrete example

Assume a fictional company, Acme Cloud, and the date is August 28, 2026. Its knowledge base contains:

• Policy version 7, effective August 1, 2026, setting a 30-day log-retention limit for EU Enterprise accounts.
• A migration memo dated July 15 that identifies plans E-2 and E-3 as still using the old 90-day setting, with a September 15 deadline.
• A support playbook dated August 20 saying that customers with an “extended retention” contract clause follow their contract instead.
• A contract template defining that clause.

The user asks which customers are affected and what support should say.

A single semantic search might return the policy and playbook but miss the migration memo. I would decompose the request into four subqueries and retrieve 30 candidates for each. The branches run in parallel, producing 120 raw results. After deduplication, perhaps 70 remain. Those 70 are reranked against the original question and the individual subquestions. The final context might contain 8 passages: the current policy, the migration memo, the exception rule, the contract definition, and supporting metadata.

If the contract exception is mentioned but its meaning is absent, the system performs an iterative lookup for the exact phrase “extended retention” plus the relevant contract type. That second hop is not speculation; it is triggered by a named bridge found in evidence.

The generated answer should say, in substance:

• Plans E-2 and E-3 are affected because the migration memo identifies them as using the old setting.
• The new 30-day limit is effective August 1, while migration is due by September 15.
• Accounts with the documented extended-retention clause are exceptions.
• Support should check the customer contract before describing the account as affected.

Each claim gets a citation to the precise document section, page, or anchored chunk. The first claim needs the migration memo, not merely the policy. The exception claim needs both the support playbook and, if necessary, the contract definition. A citation validator should reject or flag claims whose cited excerpts do not entail them.

The senior-level trade-off

I would not automatically decompose every question. Decomposition adds retrieval calls and can destroy shared context. A question containing one entity and one clear fact may be better served by a single hybrid search. I would decompose when there are multiple predicates, documents, time conditions, or an obvious join.

Likewise, increasing the candidate pool helps only when the relevant evidence exists in the index and a later stage can select it. Sending 100 mediocre passages directly to the model usually dilutes attention and increases cost. Retrieve wide, rerank, then pass a compact and diverse evidence set.

The most dangerous failure mode is stale-version mixing. The first symptom is an answer containing a current limit with an old exception, or citations whose dates contradict the answer. The fix is version-aware filtering, effective-date logic, supersession metadata, and an answer that says when sources conflict. “Newest document wins” is not enough.

What they’ll ask next

How would you evaluate this system?
Measure candidate recall at the wider retrieval stage, citation precision and coverage at the answer stage, multi-hop answer correctness, freshness accuracy, latency, and cost. Build test cases with gold source spans, stale versions, conflicting policies, and missing bridge documents.

What if two authoritative documents conflict?
Use effective dates, explicit supersession links, source hierarchy, and document scope. If the conflict remains unresolved, show both dates and state the uncertainty instead of silently blending them.

Why not just give the model more retrieved context?
More context can improve recall but also increases token cost, distracts the model, and makes citation assignment harder. The better pattern is a large candidate pool followed by reranking, deduplication, coverage checks, and a small final context.

The line to use in the room

“I treat these as complementary controls: retrieve broadly with hybrid search, decompose known hops, iterate only when evidence reveals the next key, rerank the larger pool, and cite each claim from a version-aware source span rather than citing the retrieval step wholesale.”