What is hybrid search and why is it often better than pure vector search?

How to think about it

Hybrid search runs dense vector retrieval and sparse keyword retrieval for the same query, then combines their results. It is often better than pure vector search because the two methods catch different kinds of relevance: vectors understand meaning, while keyword search protects exact names, product codes, error messages, and rare terms.

Why the combination works

Imagine an engineer asks:

How do I rotate PAYMENTS_API_KEY without downtime?

A dense embedding, which is a numeric representation of meaning, may understand that “rotate” relates to “replace,” “roll,” or “renew,” and that “without downtime” relates to blue-green deployment. That is useful. It may retrieve a page about zero-downtime credential rotation even if the page uses different wording.

But the exact string PAYMENTS_API_KEY matters too. A page about rotating DATABASE_PASSWORD is semantically similar but may be operationally useless. The system needs to find the page mentioning the actual key.

Sparse retrieval represents text using term weights, with most possible terms absent or zero. BM25 is the classic sparse ranking algorithm. It scores a document partly according to:

• how often a query term appears in that document;
• how rare that term is across the whole collection;
• how long the document is, so a 2,000-word page does not win merely by mentioning everything.

That second factor is inverse document frequency. An unusual identifier such as PAYMENTS_API_KEY carries more evidence than a common word such as “deploy.”

Dense and sparse retrieval therefore have complementary failure modes.

Dense retrieval is good at paraphrases and concepts. It can connect “cancel my subscription” with “close my account,” even when the words differ. But embedding models compress text into a fixed representation. They do not guarantee that a rare token, number, version string, or error code will dominate the representation.

BM25 is good at lexical precision. It can find ERR_PAYMENT_417, v2.7.3, or PAYMENTS_API_KEY directly. But it does not naturally understand that “rotate credentials” and “renew an API secret” may describe the same operation.

Hybrid search works because it does not ask one method to solve both problems.

What happens at query time

A typical production flow looks like this:

1. The system sends the query to a dense index and a BM25 index, usually in parallel.
2. Each retriever returns a candidate list. A candidate is a document chunk that survived this first retrieval pass.
3. The system merges the lists and fuses their rankings.
4. It may pass the merged candidates to a reranker, such as a cross-encoder that reads the query and each chunk together.
5. The application returns the best final chunks to the language model or user.

The important detail is step three. A cosine similarity score from a vector index and a BM25 score are not naturally comparable. A cosine score might be 0.82, while a BM25 score might be 14.6. Adding those raw values is meaningless because the scales, distributions, and even interpretations differ.

A common solution is Reciprocal Rank Fusion, or RRF. It uses position rather than raw score:

RRF score = 1 / (k + rank)

Here, rank starts at 1, and k is a smoothing constant. A common starting value is 60. A document appearing near the top of both lists receives a strong combined score. A document appearing in only one list still has a chance, but receives less support.

A concrete example

Suppose the two retrievers return these results for the API-key question:

Chunk	BM25 rank	Dense rank	RRF score with k = 60
D1: Rotating PAYMENTS_API_KEY with zero downtime	1	2	0.0325
D2: Using secret aliases during deployment	2	4	0.0318
D3: PAYMENTS_API_KEY version migration	3	—	0.0159
D4: Blue-green credential rotation	4	1	0.0320
D5: Rotating database credentials safely	—	3	0.0159
D6: Recovering from a payment gateway outage	—	5	0.0154

For D1, the calculation is 1 / 61 + 1 / 62, which is approximately 0.0325. It wins because it is highly ranked by both methods.

D4 is also valuable. Dense retrieval ranks it first because “blue-green credential rotation” captures the operational meaning of the question. BM25 ranks it fourth because it may not contain the exact key name. Hybrid search keeps it near the top instead of discarding it.

D3 has the exact identifier but no dense support, so it remains a plausible candidate. D5 is semantically related but lacks the payment-key terms. The final ranking gives the strongest position to documents supported by both signals, while preserving useful one-signal results.

Pure vector search might return D4 and D5 first. That would understand the general idea but could miss the page with the exact production key. Pure BM25 might return D1 and D3, but miss a well-written blue-green deployment guide that uses different terminology. Hybrid search covers both paths.

The senior-level nuance

Hybrid search is not automatically better. It is better when the query and corpus contain mixed evidence: natural language alongside exact strings.

For a small FAQ corpus containing ordinary questions such as “How do I change my billing address?”, pure vector search may be sufficient. Adding a second index creates storage, ingestion, monitoring, and tuning work without much gain.

For log search, code search, legal clauses, or technical documentation, lexical retrieval may deserve substantial weight because identifiers and numbers are important. For exploratory questions and paraphrases, dense retrieval may deserve more influence.

Candidate depth also matters. If each retriever returns only its top five results and the relevant document is ranked sixth by both, fusion cannot recover it. Hybrid search combines what the retrievers found; it does not magically search the documents they never offered. In practice, teams often retrieve a wider candidate set and let a reranker reduce it later, but the right depth depends on latency, corpus size, and evaluation results.

There is also an indexing trap. BM25 is only as good as its analyzer. If an analyzer splits, removes, or normalizes an identifier badly, the lexical signal may disappear. Fields containing error codes, SKU values, API names, and version strings often need deliberate tokenization or exact-match fields. The symptom is familiar: ordinary questions work, but searches for ERR_PAYMENT_417 return irrelevant generic pages or nothing at all.

The other common failure is duplicated evidence. If a document was chunked into six nearly identical pieces, hybrid retrieval may return all six. That looks like high retrieval confidence but gives the language model less useful context. Deduplication, document-level grouping, and metadata filters should apply consistently across both retrieval paths.

Latency is a real trade-off. Two searches issued sequentially might add roughly 20 milliseconds plus 20 milliseconds in a simple system. Issued in parallel, the retrieval portion is closer to the slower search plus network and fusion overhead, but the p95 latency can still rise when either index is under load. Hybrid search also means two indexes must stay synchronized. A stale BM25 index or stale vector index creates confusing disagreements.

What they’ll ask next

Why not normalize the two scores and add them?

You can, but normalization must be measured and maintained. BM25 and vector scores have different distributions that can change with query length, corpus composition, embedding model, and filtering. Rank-based fusion such as RRF is a robust baseline because it avoids pretending that 0.8 and 12 mean comparable things. Weighted score fusion becomes attractive when you have enough labeled queries to calibrate it.

Is hybrid search the same as reranking?

No. Hybrid search combines retrieval signals to build a candidate set. Reranking is a later stage that examines each candidate more deeply, often with a cross-encoder or another expensive model. A useful pipeline can use both: BM25 and dense retrieval for broad recall, then a reranker for final precision.

How would you evaluate and tune it?

Create a test set of real queries with judged relevant documents. Measure recall at the candidate stage, then a ranking metric such as nDCG or mean reciprocal rank at the final stage. Break results into slices: natural-language questions, identifiers, error codes, long queries, short queries, and different languages. Start with RRF, compare it against each retriever alone, and only then tune candidate depth or dense-versus-sparse weights.

Say this in the interview: Hybrid search combines dense semantic retrieval with sparse BM25 retrieval because vectors handle paraphrase while keywords preserve exact identifiers and rare terms; fusing both rankings usually improves retrieval when the corpus contains both kinds of evidence.