What is a vector database and how does it enable semantic retrieval?

How to think about it

A vector database stores embeddings, which are dense numerical representations of data, and indexes them for nearest-neighbor search. It enables semantic retrieval by embedding both documents and queries in the same vector space, then finding documents that are close by meaning rather than merely sharing words.

Why this exists

Imagine a support bot handling 10,000 handbook chunks. A customer asks, “I was billed again on my yearly plan. Can I reverse it?” The relevant article says, “Annual subscriptions can be refunded within 30 days of renewal.”

A keyword index may rank that article poorly because the wording differs. Semantic retrieval is designed for this paraphrase. It turns both texts into vectors whose positions reflect patterns learned from language. Similar ideas tend to land near one another, even when the vocabulary changes.

The important qualification is that the vector database does not understand meaning by itself. The embedding model supplies the semantic representation. The database supplies durable storage, filtering, indexing, and fast search over those representations.

The mechanism

An embedding model is a model that maps an item to a fixed-length numerical vector. For example, a text model might map each support chunk to a vector with 1,536 numbers. The numbers are not human-readable features such as “refund” or “billing”; they are coordinates in a learned space.

At ingestion time, the application splits documents into chunks, embeds each chunk, and stores the vector with its payload. The payload usually includes the original text, document ID, source URL, tenant, publication date, version, and access-control labels. A practical starting point for the support corpus might be 400-token chunks with a 50-token overlap, but chunk size must be evaluated because it affects retrieval quality.

At query time, the application embeds the user’s question with the same model and preprocessing. The database compares the query vector with stored vectors using a similarity metric such as cosine similarity. It returns the nearest k chunks, often along with their text and metadata.

A retrieval-augmented generation system then places those chunks in the language model’s context. The vector database retrieves evidence; it does not write the final answer.

What “approximate” search means

Comparing one query with every vector is exact nearest-neighbor search. That is simple, but expensive. With 100 million vectors, every query would require checking 100 million candidates before it could identify the closest ones.

Production systems therefore commonly use approximate nearest-neighbor, or ANN, indexes. ANN search deliberately examines a promising subset of the collection. It may miss the mathematically closest vector, but it can reduce search work enough to meet a latency target.

Two common index families are:

• HNSW, a graph of nearby vectors arranged in layers. Search moves through the graph toward promising regions. It often gives strong recall and low query latency, but consumes substantial memory and can take time to build.
• IVF-PQ, which assigns vectors to coarse clusters and searches selected clusters. Product quantization compresses the stored vectors, reducing memory use at the cost of some accuracy and additional tuning.

Recall is the fraction of queries for which a relevant item appears in the returned results. Increasing the search breadth generally improves recall but costs more latency, CPU, or memory. The right setting depends on the corpus and the business cost of a missed result.

Common mistake: ANN does not mean the embeddings are approximate or that the model only understands approximately. It means the database’s search procedure may skip the exact nearest neighbor to gain speed.

A concrete example

Suppose the support query is:

“I was charged for an annual renewal by mistake. Can I get a refund?”

For illustration, use three-dimensional vectors rather than real 1,536-dimensional embeddings:

• Query: [0.80, 0.60, 0.00]
• Refund policy: [0.78, 0.63, 0.00]
• Billing-email article: [0.10, 0.99, 0.00]

Cosine similarity is cosine(q, d) = (q · d) / (||q|| × ||d||). The refund policy scores about 0.999; the billing-email article scores about 0.677. The first result is closer in direction, which is what matters for cosine similarity.

A small pgvector example makes the database part concrete:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE support_chunks (
    id bigint PRIMARY KEY,
    chunk_text text NOT NULL,
    embedding vector(3) NOT NULL,
    source text NOT NULL,
    updated_at date NOT NULL
);

CREATE INDEX support_chunks_embedding_hnsw
ON support_chunks
USING hnsw (embedding vector_cosine_ops);

INSERT INTO support_chunks
    (id, chunk_text, embedding, source, updated_at)
VALUES
    (1, 'Annual plans can be refunded within 30 days of renewal.',
     '[0.78, 0.63, 0.00]', 'refund-policy', DATE '2026-01-10'),
    (2, 'Update the email address used for billing in Account Settings.',
     '[0.10, 0.99, 0.00]', 'billing-email', DATE '2026-01-10');

SELECT source, chunk_text,
       1 - (embedding <=> '[0.80, 0.60, 0.00]') AS similarity
FROM support_chunks
WHERE updated_at >= DATE '2025-01-01'
ORDER BY embedding <=> '[0.80, 0.60, 0.00]'
LIMIT 2;

Here, pgvector’s <=> operator is cosine distance, so subtracting it from 1 gives cosine similarity. The query also demonstrates an important production pattern: apply a metadata condition while performing semantic search. Real vectors would use the embedding model’s configured dimension rather than three.

A similarity score is not a probability. A score of 0.80 does not mean there is an 80 percent chance that the passage answers the question. Scores vary across embedding models, corpora, and query types. Use labeled evaluation data to select a useful threshold.

Choosing the metric

Metric	Useful when	Important caveat
Cosine similarity	Text embeddings where vector direction represents relatedness	Normalization and model guidance matter
Dot product	The embedding model is trained or configured for dot-product scoring	Magnitude affects ranking
L2 distance	Magnitude and straight-line distance are meaningful for the model	Scale can dominate if vectors are not normalized

For normalized vectors, cosine similarity and dot product produce the same ranking. L2 distance also produces an equivalent ranking in that specific case. Do not choose a metric because it sounds faster. Match the metric to the embedding model and verify it with retrieval tests.

The production pattern

A robust retrieval pipeline often retrieves more candidates than it finally uses. For example, a support bot might retrieve 20 chunks, apply a cross-encoder reranker to the candidates, and pass the best 5 to the language model. Reranking can improve precision because it reads the query and candidate text together, but it adds inference cost and latency.

Metadata filters are equally important. Filter by tenant, product, language, document version, publication status, or permissions. If a system retrieves 100 candidates and only two belong to the requesting customer, post-filtering may leave too little useful context. Whether filtering happens before or during ANN search depends on the database and index, so test that behavior rather than assuming it.

Semantic search should usually be hybrid. A vector search is good at “reverse a yearly charge” versus “refund an annual renewal.” Lexical search is often better for an exact identifier such as ERR-1042, a product SKU, a person’s name, or a legal clause. Combining lexical and vector results, then reranking them, handles both cases.

Trade-offs and failure modes

A vector database is the wrong tool for a primary-key lookup, a financial transaction, a relational join, or a GROUP BY report. If the collection contains only a few hundred stable documents, brute-force similarity search may be simpler than operating a separate database and index. Vector systems earn their keep when nearest-neighbor retrieval, filtering, persistence, and scale are central requirements.

Changing the embedding model is also a migration. A vector from model A and a vector from model B are not comparable merely because both have 1,536 dimensions. Re-embed the corpus, or maintain separate collections during a controlled migration.

A common failure appears as fluent but unsupported answers. The first symptom is usually visible before the language model runs: the top retrieved chunks are broadly related but omit the exact refund window, exception, or date. Causes include poor chunk boundaries, stale documents, the wrong embedding model, an overly aggressive ANN setting, or a missing metadata filter. Log the query, model version, retrieved IDs, scores, filters, and raw text. Inspect retrieval independently from generation.

Never treat semantic relevance as authorization. A highly similar document from another tenant is still forbidden context.

What they’ll ask next

Why not use a normal keyword search engine?
Keyword search is excellent for exact terms, identifiers, and rare names. Vector search handles paraphrases and related concepts. Most serious systems use hybrid retrieval because each method covers the other’s blind spots.

When would you choose HNSW over IVF-PQ?
Choose HNSW when low-latency, high-recall online search justifies its memory use. Consider IVF-PQ when the collection is much larger or memory reduction is important. Benchmark recall, p95 or p99 latency, index-build time, update behavior, and cost on the actual workload.

How do you evaluate semantic retrieval?
Create labeled query-to-document judgments and measure whether a relevant chunk appears in the first k results, using metrics such as Recall@k, MRR, or nDCG. Then evaluate the complete system for grounded answers, latency, and cost. A good nearest-neighbor score is not enough if the generated answer still cites stale or irrelevant evidence.

Say this in the interview: A vector database stores embeddings and uses nearest-neighbor indexes to retrieve semantically related content, while the embedding model supplies the meaning and production systems add filters, hybrid search, and reranking for precision and control.