Vector databases

A vector DB is a database optimized for one query: “given this query vector, find the K nearest stored vectors.” A small RAG demo can do this with a numpy array and brute-force cosine. A production system can’t — not over 10 million chunks with a 50ms latency budget. That’s where vector databases come in.

When you actually need one

The honest threshold: fewer than ~100,000 vectors and you don’t need one. A numpy array on a single machine handles brute-force cosine over 100k 1536-dim vectors in well under a second. You’re better off keeping things simple.

You graduate to a vector DB when:

1. You have millions to billions of vectors.
2. You need sub-second query latency under load.
3. You need persistence across restarts (numpy in RAM doesn’t qualify).
4. You need metadata filters alongside the vector search.
5. You need it to be queryable from multiple services.

If you check none of these, ship the numpy array. If you check three or more, pick a real vector DB.

The landscape

Several systems compete; each has a niche:

What’s inside — the ANN index

Brute-force search compares the query against every stored vector. Over 5,000 vectors that’s instant. Over 10 million, on every query, it collapses — ten million distance calculations to answer one question. To go faster you need a sublinear index: one that finds the nearest neighbors without ever looking at most of the data.

The two you’ll meet are HNSW and IVF. Both are Approximate Nearest Neighbor (ANN) indexes — “approximate” because they accept a tiny chance of missing the true nearest neighbor in exchange for being orders of magnitude faster. Well-tuned, they return the correct top results 95–99% of the time.

HNSW — a map with express lanes

The trick behind HNSW (Hierarchical Navigable Small World) is the one your sat-nav uses. To cross a country you don’t drive every local street — you take the motorway most of the way, exit near your destination, and only then navigate the small roads. HNSW builds exactly those layers of road on top of your vectors.

How it builds the layers. Every vector goes into the bottom layer, Level 0 — the full map, where each point is wired to its handful of nearest neighbors by short edges. Then each vector flips a biased coin: most stay at Level 0, a few are promoted up to Level 1, and far fewer again to Level 2. So each layer up is a sparser sample of the one below — fewer nodes, and the links between them stretch much further. Level 2 is the motorway; Level 0 is the side streets.

How it connects the dots. Inside every layer, each node links to its ~M closest neighbors. Because most links are short but a few reach far, any node can get to any other in only a few hops — the “small world” property (the same reason you’re a handful of handshakes from anyone on earth).

How a query searches. Enter at the single top node and greedily hop to whichever neighbor is closest to the query. When no neighbor gets you any closer, drop down a layer — same spot on the map, but now with more nearby nodes to choose from — and keep going. The top layers cover enormous distance per hop (big jumps); the bottom layer does the careful final search (small steps). A few dozen hops replace millions of comparisons.

IVF — split the space into cells

IVF (Inverted File index) takes the other path. Before any query it clusters all the vectors — k-means into, say, a few thousand cells, each with a centroid. A query then compares against the centroids, picks the few nearest cells, and searches only those, skipping every vector in the cells it didn’t pick. Lower memory than HNSW and faster to build, but recall drops when the true neighbor sits just across a cell boundary you didn’t probe. It’s common at billion-vector scale, where HNSW’s in-RAM graph gets expensive.

Most managed vector DBs default to HNSW, and you usually don’t pick the index at all — you pick the system, and it picks the index for you. Tuning the knobs (ef_construction, M, nlist) is a fine-tuning job for once you have real traffic.

That graph descent is how the index finds candidates fast; the final ranking is then just plain similarity over those candidates — exactly what the insert/query example further down computes, score by score.

Metadata filters — the unsung feature

A pure ANN query is “find the K closest vectors.” Real apps need “find the K closest vectors AMONG documents the current user is allowed to see, written after 2024, in category=help”. That’s a metadata filter combined with the vector search.

results = vector_db.query(
    vector=query_embedding,
    top_k=10,
    filter={
        "user_org_id": current_user.org_id,
        "category": "help",
        "created_at": {"$gte": "2024-01-01"},
    },
)

Every vector DB worth using supports this. Test it on your queries — some systems filter post-ANN (retrieve 100 candidates, filter, return 10), others filter during the ANN walk. Performance differs dramatically when filters are very selective (e.g., “only this one user’s documents”).

The API shape — insert and query

Most vector DBs converge on a similar API: insert vectors with IDs and metadata, query by vector with optional filters. Here’s the shape you’ll write against, using a mock client:

import math

# A toy in-memory "vector DB" — same API shape as pgvector / Pinecone / Qdrant.
class MockVectorDB:
    def __init__(self):
        self.records = []   # list of dicts: id, vector, metadata

    def insert(self, id, vector, metadata):
        self.records.append({"id": id, "vector": vector, "metadata": metadata})

    def _cosine(self, a, b):
        dot = sum(x*y for x, y in zip(a, b))
        na = math.sqrt(sum(x*x for x in a))
        nb = math.sqrt(sum(y*y for y in b))
        return dot / (na * nb) if na and nb else 0.0

    def query(self, vector, top_k=5, filter=None):
        # Filter first if the user asked for it
        candidates = self.records
        if filter:
            candidates = [r for r in candidates
                          if all(r["metadata"].get(k) == v for k, v in filter.items())]
        # Score each, return top_k
        scored = [(r, self._cosine(vector, r["vector"])) for r in candidates]
        scored.sort(key=lambda x: -x[1])
        return [{"id": r["id"], "score": s, "metadata": r["metadata"]}
                for r, s in scored[:top_k]]

# Mock embedder — bag-of-words over a tiny vocab
def embed(text):
    text = text.lower()
    vocab = ["python", "sql", "billing", "refund", "login", "password", "ml"]
    return [1.0 if w in text else 0.0 for w in vocab]

db = MockVectorDB()
db.insert("doc1", embed("How do I reset my password?"),    {"category": "auth"})
db.insert("doc2", embed("ML model training with Python"),  {"category": "tech"})
db.insert("doc3", embed("Refund policy and billing"),      {"category": "billing"})
db.insert("doc4", embed("Login with SSO"),                 {"category": "auth"})
db.insert("doc5", embed("Python SQL connector"),           {"category": "tech"})

q = embed("I forgot my password — how to reset?")
results = db.query(q, top_k=2, filter={"category": "auth"})
print("Auth-only matches:")
for r in results:
    print(f"  {r['id']:5s}  score={r['score']:.3f}  {r['metadata']}")

results = db.query(q, top_k=3)
print("\nNo filter, top 3:")
for r in results:
    print(f"  {r['id']:5s}  score={r['score']:.3f}  {r['metadata']}")

Auth-only matches:
  doc1   score=1.000  {'category': 'auth'}
  doc4   score=0.000  {'category': 'auth'}

No filter, top 3:
  doc1   score=1.000  {'category': 'auth'}
  doc2   score=0.000  {'category': 'tech'}
  doc3   score=0.000  {'category': 'billing'}

Trace it. The query embeds the password-reset question to the same vector as doc1, so doc1 scores a perfect 1.000 and every unrelated doc scores 0.000. The first call’s filter={"category": "auth"} is the unsung feature: doc2 (tech) outranks nothing here because it was filtered out before scoring — only doc1 and doc4 survive the filter. The shape is universal: insert(id, vector, metadata) + query(vector, top_k, filter). What varies is the implementation behind the API — HNSW graphs in Qdrant, IVF lists in Faiss, HNSW-on-Postgres in pgvector, custom storage in Pinecone.

pgvector — what production looks like

Real pgvector SQL has the shape:

-- Schema with a vector column
CREATE TABLE chunks (
  id        BIGSERIAL PRIMARY KEY,
  doc_id    INT,
  chunk_text TEXT,
  org_id    INT,
  embedding VECTOR(1536)
);

-- ANN index (HNSW)
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- Query: nearest 10 chunks, scoped to the user's org
SELECT id, chunk_text, embedding <=> '[0.1, 0.2, ...]' AS distance
FROM chunks
WHERE org_id = 42
ORDER BY embedding <=> '[0.1, 0.2, ...]'
LIMIT 10;

<=> is the cosine-distance operator pgvector adds. The WHERE clause filters first when the index makes that efficient. Joins, transactions, foreign keys — all normal Postgres. No separate ops layer.

Avoid these traps

A handful of consistent mistakes:

• Embedding dimension mismatch. If you ever change embedding models, you must re-embed every stored vector. Mixing 1536-dim OpenAI vectors and 1024-dim Cohere vectors in the same index = silent wrong results.
• No metadata for filtering. People insert just (id, vector) and then realize they can’t do row-level security. Always store the org/user/tenant id alongside.
• Premature optimization. Tuning HNSW parameters before you have real queries is wasted work. Profile first.
• Forgetting to update. Documents change, get deleted. Your vector DB needs to reflect that. Bake “delete + reinsert” into your indexing pipeline; don’t let stale vectors accumulate.

Hybrid search

Pure vector search misses exact matches (product IDs, error codes). Pure keyword search misses synonyms. Hybrid search runs both and merges — typically with Reciprocal Rank Fusion (RRF). The RAG-basics and advanced-RAG lessons cover this. Most vector DBs now ship hybrid out of the box; check before adding a separate search system.

In one breath

• A vector DB answers one query fast: “given this vector, find the K nearest stored vectors” — at a scale a numpy array can’t.
• Below ~100k vectors, a numpy array is enough; graduate to a DB for millions of vectors, sub-second latency, persistence, filters, or multi-service access.
• ANN indexes (HNSW, IVF) are sublinear and 95–99% recall — HNSW is a layered “express-lane” graph (big jumps up top, fine steps at the bottom); IVF splits space into cells and probes a few.
• Metadata filters turn “K nearest” into “K nearest that this user may see” — store the org/user id alongside every vector.
• The boring winner is usually pgvector: vector types + HNSW on the Postgres you already run; move to a dedicated system only at real scale.

Quick check

Next

You’ve seen embeddings as concepts and vector DBs as infrastructure. The next lesson is RAG basics — wiring chunks, embeddings, retrieval, and an LLM into the pattern that grounds models in your data.