Advanced RAG — beyond top-k

You shipped a RAG prototype. It worked on the demo questions. Then real users showed up and recall fell off a cliff — exact-match queries missed, paraphrased questions retrieved the wrong page, and the chunks that did match were missing the surrounding context the model needed to actually answer. This is where naive embed(query) -> top-k cosine runs out of room.

The biggest single fix in the last two years has a name now: Anthropic’s contextual retrieval. It addresses the root problem that almost every other RAG trick is working around — that chunks lose context the moment you cut them out of their document.

Contextual retrieval — the headline upgrade

Naive chunking destroys context. A chunk that reads:

“The company’s revenue grew by 3% over the previous quarter.”

…is meaningless on its own. Which company? Which quarter? Embedding it puts it in a generic “small percentage growth” region of the vector space — far from a user query like “ACME Q2 2024 revenue growth”.

Anthropic’s fix: before embedding, prepend a 50-100 token LLM-generated summary that situates the chunk in its document. The chunk becomes:

“This chunk is from an SEC filing on ACME Corp’s performance in Q2 2024; the previous quarter’s revenue was $314 million. The company’s revenue grew by 3% over the previous quarter.”

Now embed that. The chunk lives in the right region of vector space, and BM25 picks up the company name and time period for free.

The contextualizing step, with prompt caching

The “obvious” cost worry: an LLM call per chunk per document is expensive. The trick: prompt caching makes the whole document the cached prefix; only the chunk varies between calls. Anthropic measured the one-time cost at roughly $1.02 per million document tokens for contextualization on Haiku-class models.

# Sketch of the contextualizing step. Run once per document, per chunk.
# In production: pip install anthropic
# from anthropic import Anthropic
# client = Anthropic()

CONTEXTUALIZE_PROMPT = """\
<document>
{whole_document}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>

Please give a short succinct context to situate this chunk within the
overall document for the purposes of improving search retrieval of the
chunk. Answer only with the succinct context and nothing else."""

# response = client.messages.create(
#     model="claude-haiku-4.6",
#     max_tokens=200,
#     system=[
#         {"type": "text",
#          "text": CONTEXTUALIZE_PROMPT.split("<chunk>")[0],
#          "cache_control": {"type": "ephemeral"}},  # cache the whole-doc prefix
#     ],
#     messages=[{"role": "user", "content": chunk}],
# )
# context = response.content[0].text
# contextualized = f"{context}\n\n{chunk}"
# embed(contextualized)  # AND index in BM25

The contextualized chunk is what you embed and what you index in BM25. Both retrievers benefit.

Results, in Anthropic’s numbers

These are unusually big numbers for a single technique. If your RAG system handles documents long enough to be chunked, this is the first upgrade to try.

The full modern pipeline

Indexing (offline, once per document):
  doc -> chunks
       -> for each chunk: LLM(whole doc cached, chunk varies)
                          -> contextualized chunk
       -> embed contextualized chunks -> vector index
       -> index contextualized chunks -> BM25

Query time:
  query -> dense retrieval  -> top-150
        -> sparse (BM25)    -> top-150
        -> fuse with RRF    -> top-50
        -> cross-encoder rerank -> top-20
        -> (optional) swap chunks for parent docs
        -> stuff into LLM prompt

The earlier “4-stage pipeline” is now stage 2-4 of this. Contextual retrieval is stage 0: an offline cost you pay once per document to make every downstream stage work better.

Most teams ship the dense-only path and stop. Each added stage in this diagram earns its keep — contextualizing has the biggest absolute gain, re-ranking is the safest second add, query rewriting and parent-doc retrieval round it out.

Hybrid retrieval: BM25 + dense + RRF (the partner to contextual retrieval)

Contextual retrieval lifts both retrievers; hybrid retrieval makes sure you’re using both. Dense embeddings are good at meaning. They are bad at exact strings. Try retrieving a chunk that mentions ERR_TLS_CERT_EXPIRED with a pure-semantic search — the model embeds the error code as a generic “error-shaped” vector and you get back chunks about network errors in general. BM25 (the classic TF-IDF descendant) nails this because it indexes literal tokens.

Reciprocal rank fusion is the standard way to merge two ranked lists. It only uses ranks, not raw scores, so you don’t have to calibrate BM25 and cosine to the same scale:

RRF(d) = Σ over retrievers i of  1 / (k + rank_i(d))

with k = 60 as the empirical default. Documents that rank high in either list bubble up; documents that rank high in both dominate.

# Reciprocal Rank Fusion over two ranked retrievers.
# In production, replace mock_bm25 / mock_dense with Elasticsearch +
# pgvector / Pinecone / Weaviate / Qdrant calls.

docs = [
    "TLS certificate expired error ERR_TLS_CERT_EXPIRED in browser",  # 0
    "How to debug network timeouts in production systems",            # 1
    "Renew SSL certificate using certbot on Ubuntu servers",         # 2
    "TLS handshake fundamentals and certificate chains",             # 3
    "Cooking recipes for slow-cooker chili and stew",                # 4
]

def mock_bm25(query, docs):
    # BM25 stand-in: rank by literal keyword overlap (great at exact tokens)
    q_terms = set(query.lower().split())
    scored = [(i, len(q_terms & set(d.lower().split()))) for i, d in enumerate(docs)]
    scored.sort(key=lambda x: -x[1])
    return [i for i, _ in scored]            # ranked doc indices

def mock_dense(query, docs):
    # Dense stand-in: a fixed semantic ranking for the demo
    semantic_rank = {0: 2, 1: 3, 2: 1, 3: 0, 4: 4}
    return sorted(range(len(docs)), key=lambda i: semantic_rank[i])

def rrf(ranked_lists, k=60, top_n=3):
    scores = {}
    for ranking in ranked_lists:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])[:top_n]

query = "fix ERR_TLS_CERT_EXPIRED error"
bm25_ranked = mock_bm25(query, docs)
dense_ranked = mock_dense(query, docs)

print("BM25 ranking: ", bm25_ranked)   # exact token wins -> doc 0 first
print("Dense ranking:", dense_ranked)  # semantics -> doc 0 only middling
print("Fused top-3 (doc id, RRF score):")
for doc_id, score in rrf([bm25_ranked, dense_ranked]):
    print(f"  doc {doc_id}  {score:.4f}  {docs[doc_id]}")

BM25 ranking:  [0, 1, 2, 3, 4]
Dense ranking: [3, 2, 0, 1, 4]
Fused top-3 (doc id, RRF score):
  doc 0  0.0323  TLS certificate expired error ERR_TLS_CERT_EXPIRED in browser
  doc 3  0.0320  TLS handshake fundamentals and certificate chains
  doc 2  0.0320  Renew SSL certificate using certbot on Ubuntu servers

Watch doc 0. BM25 ranks it first because it contains the literal error code; the dense retriever only ranks it third because semantically it looks like generic TLS content. Neither retriever alone is confident — but RRF rewards a doc that ranks well in either list, so doc 0 fuses to the top while the off-topic chili recipe (doc 4) never surfaces. In real corpora, hybrid + RRF reliably beats either retriever alone by 10-20 points of recall@10, for one extra index and a few milliseconds. Always-on.

Query rewriting: HyDE and multi-query (complement to the above)

Contextual retrieval fixes the chunk side of the question/answer mismatch. Query rewriting fixes the query side. They’re additive.

Embedding the user’s question and matching it against embedded answer chunks is asymmetric — questions and their answers don’t embed near each other. “What’s the refund window?” and the policy chunk “Customers may return purchases within 30 days” share zero words and live in different regions of the vector space.

Two fixes:

HyDE (Hypothetical Document Embeddings). Ask a cheap LLM to hallucinate an answer to the query, then embed that and use it as the retrieval vector. The hallucinated answer doesn’t need to be factually right — it just needs to be answer-shaped. It will embed near the real answer.

Multi-query. Have the LLM generate 3-5 paraphrases of the query and retrieve for each, then union the results (often via RRF). Helps when the user’s phrasing is unusual.

Both add a cheap LLM call (use gpt-5-mini or claude-haiku-4.6) before retrieval. Add 200-400ms. Worth it for hard queries; skippable for obvious lookups.

Re-ranking with cross-encoders

A retriever returns ~50 candidates by embedding similarity. A cross-encoder then jointly encodes [query, candidate] pairs and produces a precise relevance score. It can’t precompute (the input includes the query), which is why it can’t replace retrieval — but re-ranking 50 candidates is cheap, and the precision gain is huge.

Production picks:

• Cohere rerank-3.5 — fast hosted, multilingual, currently the default choice for English+global content.
• voyage-rerank-2 — strong on technical / long-context domains.
• BGE-reranker-v2-m3 — open-weights, self-hostable, competitive.

# In production:
# from cohere import Client
# co = Client(api_key=...)
# results = co.rerank(
#     model="rerank-3.5",
#     query=user_query,
#     documents=[c.text for c in candidates],
#     top_n=10,
# )
# reordered = [candidates[r.index] for r in results.results]

Recall@10 typically jumps 5-15 points after re-ranking. The cost is ~$1-2 per 1000 reranks and 100-300ms latency. Default-on for any serious system.

Parent-document retrieval (a different angle on the same problem)

Contextual retrieval makes each chunk more self-sufficient at embedding time. Parent-document retrieval takes the opposite approach: keep chunks small for matching, but swap them for their parent document at generation time. The two techniques are complementary; you can use both.

Here’s the tension parent-document solves: small chunks (200-400 tokens) retrieve well because they’re focused; large chunks (1500-3000 tokens) generate well because the model has context. You don’t have to choose.

Parent-document retrieval indexes small chunks but stores a pointer to the larger parent (a section, a page, a doc). At query time you retrieve via the small chunks, then swap them for their parents before sending to the LLM. You get matching precision and generation context.

# Conceptual structure
chunks = [
    {"id": "c1", "text": "...30 days...", "parent_id": "p1"},
    {"id": "c2", "text": "...refund...",  "parent_id": "p1"},
]
parents = {"p1": "Full refund policy section — 1200 tokens of context..."}

# 1. Retrieve at chunk level (precision)
hits = vector_search(query, chunks)
# 2. Expand to parents (context)
unique_parents = {chunks[h].parent_id for h in hits}
context = [parents[pid] for pid in unique_parents]

LangChain and LlamaIndex both ship this out of the box. You can build it on top of any vector DB by storing parent_id as metadata.

Metadata filtering: the force multiplier

Every chunk should carry metadata: source, doc_type, team, updated_at, language. A vector DB that supports filtered search (pgvector, Qdrant, Pinecone, Weaviate all do) lets you constrain before similarity scoring:

# Pseudocode for a metadata-aware search
results = vector_db.search(
    query_vector=qvec,
    filter={"team": "billing", "updated_at": {">": "2026-01-01"}},
    top_k=20,
)

This cuts the search space, kills stale-document hallucinations, and enforces access control all at once. The cheapest way to improve a RAG system is usually “filter to the right 5% of the corpus first.”

In one breath

• Contextual retrieval is the headline upgrade: prepend a short LLM-written summary to each chunk before embedding/indexing so it carries its document context — prompt caching makes it cheap (~$1/M doc tokens), cutting retrieval failures ~49% (67% with reranking).
• Hybrid retrieval runs dense (meaning) + BM25 (exact tokens) and fuses with RRF, which uses only ranks — no score calibration needed.
• Query rewriting fixes the question side: HyDE retrieves with an answer-shaped (even hallucinated) vector; multi-query unions paraphrases. Gate them to the hard queries.
• A cross-encoder re-ranker rescoring ~50 candidates is the safest high-ROI add after contextualizing.
• Parent-document retrieval indexes small chunks for precision, serves the big parent for context — and retrieval quality is not evidence integrity, so govern who can write to the corpus.

Quick check

Next

You’ve now got a retrieval pipeline that holds up under real traffic. The next problem is knowing whether it actually works — which is what RAG evals are for.