Sentence embeddings: SBERT

At 3 a.m., a customer searches your help centre for:

Why was my card charged twice?

The useful article says:

Duplicate debit-card transactions usually disappear within five business days.

A keyword search may miss it. The words do not line up, even though the meanings do. So you try a language model. It understands the pair beautifully.

Then you remember the help centre has one million articles and snippets.

To find the best match, you would need to ask the model one million questions:

• Does this query match article 1?
• Does it match article 2?
• What about article 3?

That is accurate in the same way that reading one million essays is accurate. It is also a poor search engine.

Embeddings as vectors explained what a sentence embedding is: a point in a numerical space whose location is meant to capture meaning. This lesson is about getting useful sentence vectors without making the model inspect every query-document pair.

The key idea is SBERT, or Sentence-BERT. It uses a bi-encoder, which encodes two sentences independently, instead of a cross-encoder, which reads the two sentences together. The first is cheap and indexable. The second is slower and usually more accurate.

Production systems commonly use both.

The cross-encoder sees the relationship

A cross-encoder receives a pair of texts as one input and produces a score for that pair. With BERT, the input is conceptually:

[CLS] query [SEP] document [SEP]

The important detail is shared attention. Tokens in the query can attend directly to tokens in the document, and vice versa. The model can therefore distinguish “refund available” from “refund not available,” or “Apple support” from “support for apples.”

It builds a fresh joint representation for every pair, then predicts relevance or similarity. That joint reading makes it strong, but the document representation depends on the query beside it, so it cannot be prepared ahead of time.

For one million documents, one query requires one million cross-encoder evaluations. Ten queries require ten million. The count grows with the product of queries and documents.

If one pair took 10 milliseconds on one worker:

1,000,000 × 0.010 seconds = 10,000 seconds

That is about 2.78 hours for one query on that worker. The figure is illustrative, and batching or GPUs change wall-clock time, but every candidate still has to pass through the model for every query.

A cross-encoder produces a score, not a reusable document vector. You can cache particular query-document scores, but there are too many possible pairs for that to solve general search.

The bi-encoder encodes once

A bi-encoder independently encodes the two inputs using two towers; the towers may share weights, as in SBERT, or use separate weights for asymmetric query and document roles. It sends the query and candidate through separate paths, each returning a fixed-size vector.

At serving time:

1. Encode every document once.
2. Store the document vectors in a vector index.
3. Encode a new query once.
4. Search for nearby document vectors.

The model loses token-level interaction between query and document, but gains the property search needs: a document vector is reusable.

The comparison is often cosine similarity:

cosine(a, b) = (a · b) / (||a|| × ||b||)

The dot is the dot product and the double bars indicate vector length. Cosine measures the angle between vectors: 1 means the same direction, 0 means orthogonal, and negative means opposite directions.

Consider this toy two-dimensional space:

import numpy as np

query = [1.0, 0.0]
cands = {
    "a dog barks": [0.9, 0.1],
    "the sky is blue": [0.1, 0.9],
    "a puppy yaps": [0.95, 0.05],
}

def cos(a, b):
    a, b = np.array(a), np.array(b)
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

for c in sorted(cands, key=lambda c: cos(query, cands[c]), reverse=True):
    print(f"{cos(query, cands[c]):.2f}  {c}")

N = 1_000_000
print(f"\n{N:,} candidates: cross-encoder = {N:,} model runs/query; bi-encoder = {N+1:,} encodings + fast vector search")

1.00  a puppy yaps
0.99  a dog barks
0.11  the sky is blue
1,000,000 candidates: cross-encoder = 1,000,000 model runs/query; bi-encoder = 1,000,001 encodings + fast vector search

The vectors are invented, but the arithmetic illustrates the point: “a puppy yaps” ranks above “a dog barks” without sharing the query’s words. The document encodings happen offline and are reused; each later query adds one query encoding, not another million document passes.

A vector index can scan every vector exactly or use approximate nearest-neighbour search, which explores promising regions faster but may miss the mathematical nearest item. Measure retrieval quality rather than assuming it.

Operationally:

• Query and document vectors must come from the same embedding space, with matching dimensions.
• Apply required query and passage prefixes consistently.
• Re-embedding with a new model requires rebuilding or versioning the index.
• Cosine, dot-product, and Euclidean search are not interchangeable unless normalization makes them equivalent.

A similarity score is not a probability. 0.82 does not mean “82 percent relevant”; its useful range depends on the model, data, and query type.

Why ordinary BERT vectors are not enough

BERT was trained mainly for contextual token understanding and masked-token prediction. Those objectives do not directly teach it that these sentences should be neighbours:

The delivery arrives tomorrow.
Your package should be here the next day.

Averaging BERT’s token vectors can produce a sentence representation, but it is not automatically a good similarity representation. SBERT makes sentence-level geometry part of the training task.

A siamese network uses two copies of an encoder with shared weights. The copies process two sentences independently, then compare their vectors to calculate a loss. For example, “A refund takes five business days” and “Refunds can take five working days” should be close, while “Reset your password from the account settings” should be farther away.

A contrastive objective pulls positive examples together and pushes negatives apart. A triplet objective supplies an anchor, a positive, and a negative, asking for:

similarity(anchor, positive) > similarity(anchor, negative)

The original SBERT work used sentence-pair and triplet-style setups, including natural language inference (NLI) data. NLI labels whether one sentence entails, contradicts, or is neutral toward another. The exact recipe varies by model.

Modern training often uses in-batch negatives. In a batch of 64 query-positive pairs, the paired passage is the positive for each query; the other 63 passages serve as negatives, at least provisionally. One batch therefore supplies many comparisons.

The quality of negatives matters. Unrelated weather sentences are easy negatives. Hard negatives are plausible but wrong, such as “How do I close my account?” versus “How do I temporarily freeze my account?” They force the model to learn distinctions users actually need. A cross-encoder can help find hard negatives, but “looks similar” is not the same as “answers the question.”

After training, the model can encode a sentence without knowing its future comparison partner. That is the serving advantage, not a guarantee of universal meaning. A model trained on customer-support pairs may be excellent for support retrieval and poor at legal clauses or scientific claims.

Retrieve first, rerank second

The practical compromise is retrieve-and-rerank:

1. A bi-encoder retrieves a shortlist from the full corpus.
2. A cross-encoder scores the query against each shortlisted document jointly.
3. The system returns the reranked results or passes them to RAG.

With one million documents and a shortlist of 100, the cross-encoder evaluates:

1,000,000 ÷ 100 = 10,000

times fewer pairs than a full scan.

The shortcut has a hard boundary. If the correct article ranks 173rd in bi-encoder results, a reranker that sees only the top 100 cannot rescue it. A reranker improves the shortlist; it is not a search party for discarded documents.

Measure the stages separately:

• Retrieval recall at k: whether the relevant document appears in the first k results.
• Reranking quality: whether the cross-encoder moves relevant results toward the top of that shortlist.

If the correct article appears in the first 100 results for 92 of 100 questions, recall at 100 is 92 percent. No reranker can fix the other eight unless retrieval improves or the shortlist grows.

Which search shape should you choose?

Hybrid retrieval combines lexical search such as BM25 with vector search. BM25 is strong for ERR_CONN_RESET, invoice numbers, drug names, and API flags; a bi-encoder is stronger at paraphrases. For RAG, a semantically close document that answers a different question can be worse than an exact match, so evaluate retrieval rather than relying on model size.

What breaks first in production

The relevant document never enters the shortlist

This is a first-stage recall failure. The embedding model may miss domain language, chunking may be poor, an exact identifier may need lexical search, or a metadata filter may have removed the document.

Test recall at several shortlist sizes. Inspect chunks and filters, add hard negatives, try a domain-suitable model, and add hybrid retrieval. A better cross-encoder cannot help a document it never receives.

The index and encoder disagree

A model change can alter vector dimension, normalization, or the learned space. Even matching dimensions do not make old document vectors compatible with new query vectors.

Store the model version beside the index. Re-embed when changing models, and compare old and new indexes on the same evaluation set instead of silently mixing vectors.

The input format is wrong

Some models require different query and passage prefixes or instructions. Omitting them changes the inputs from what training saw; adding an undocumented prefix can hurt other models.

Treat preprocessing as part of the model. Record the exact functions used for indexing and querying, and apply them consistently.

Similarity is high but relevance is wrong

A query such as “cancel my subscription” may retrieve “pause your subscription.” Add hard negatives that express this distinction, rerank with a cross-encoder, and filter by product, region, date, or account type. Also inspect chunk size: a whole page may be broadly similar while hiding the useful paragraph.

The limitation worth remembering

SBERT makes sentence comparison scalable by dropping token-level interaction during the first stage. A bi-encoder creates reusable representations but cannot fully inspect every query phrase against every document phrase while producing one vector. A cross-encoder can perform that inspection, but repeats it for every pair.

Use a bi-encoder for broad retrieval, a cross-encoder for a small candidate set or pair-level precision, and both when missing a candidate is costly and ordering the first results matters. Training examples, negatives, chunk boundaries, index freshness, filters, and evaluation queries often matter as much as the model.

In one breath

• A cross-encoder reads a query and document together. It is accurate, but searching N documents requires N evaluations and produces no reusable document vector.
• A bi-encoder encodes each sentence independently into a reusable vector for indexed similarity search.
• SBERT trains this geometry with shared-weight branches and objectives that pull useful matches together and separate confusing negatives.
• Cosine similarity is a geometric score, not a relevance probability.
• The production pattern is retrieve then rerank.
• A reranker cannot recover a relevant document that first-stage retrieval never returned.

Quick check

Next

These vectors are what vector databases store and what RAG retrieves. For the geometry behind an embedding, see embeddings as vectors. For the broader retrieval problem, keep recall and reranking quality separate.