Embeddings as vectors

An embedding is a list of numbers that represents a piece of text in a way that captures its meaning. “I love dogs” and “I adore puppies” embed near each other; “I love dogs” and “Python is a typed language” embed far apart. That’s the whole pitch. Once you internalize that texts become points in high-dimensional space, everything that follows — semantic search, clustering, recommendation — is geometry.

The mental model

Imagine a 2D plane. Place “cat” near “kitten” and far from “bicycle”. Place “dog” near “puppy” but also near “cat” (both pets). Place “Python” far from all the animals, but near “Java” (both programming languages). That’s a 2D embedding space.

Real embedding models output hundreds to thousands of dimensions — typically 384, 768, 1024, 1536, or 3072. The geometry is the same; you just can’t draw it.

# A tiny made-up embedding model that maps words into 2D.
# Real embeddings come from a neural net; the SHAPE is the same.
embeddings = {
    "cat":    (0.8, 0.6),
    "kitten": (0.85, 0.55),
    "dog":    (0.7, 0.7),
    "puppy":  (0.75, 0.72),
    "python": (-0.6, 0.2),
    "java":   (-0.55, 0.15),
    "bicycle": (-0.1, -0.8),
}

import math

def euclidean(a, b):
    return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))

# Nearest neighbours of 'cat'
query = embeddings["cat"]
dists = [(word, euclidean(query, vec)) for word, vec in embeddings.items() if word != "cat"]
dists.sort(key=lambda x: x[1])
for word, d in dists:
    print(f"  {word:9s} distance {d:.3f}")

  kitten    distance 0.071
  puppy     distance 0.130
  dog       distance 0.141
  java      distance 1.423
  python    distance 1.456
  bicycle   distance 1.664

Read the distances top to bottom. The other animals sit within 0.15 of “cat”; the programming languages are ten times farther at ~1.4; the bicycle is farther still. That’s exactly what an embedding does — for sentences, paragraphs, and documents, in many more dimensions.

The same idea drawn out: meaning becomes position. Below, a handful of words are placed so that neighbours are near and unrelated words are far. “cat” is the query, and the lines run to its three nearest points — the very ones the code ranked first. That single move, “find the closest vectors to the query,” is the entire engine behind semantic search and, soon, RAG.

Three distance metrics

Once two texts are vectors, you need a way to ask “how similar are they?”. Three common metrics:

Cosine similarity

The cosine of the angle between two vectors. Range: [-1, 1]. 1 = same direction (similar), 0 = orthogonal (unrelated), -1 = opposite. Ignores magnitude — only direction matters.

cos_sim(a, b) = dot(a, b) / (||a|| * ||b||)

Euclidean (L2) distance

The straight-line distance. Sensitive to magnitude. Closer = more similar.

euclid(a, b) = sqrt(sum((a_i - b_i) ** 2))

Dot product

Sum of a_i * b_i. If vectors are already normalized (length 1), dot product equals cosine similarity — and it’s faster because it skips the normalization step.

dot(a, b) = sum(a_i * b_i)

import math

def cosine(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a))
    nb = math.sqrt(sum(y*y for y in b))
    return dot / (na * nb) if na and nb else 0.0

def euclid(a, b):
    return math.sqrt(sum((x-y)**2 for x, y in zip(a, b)))

a = [1, 2, 3]
b = [2, 4, 6]      # same direction, double the magnitude
c = [-1, -2, -3]   # opposite direction

print(f"cosine(a, b) = {cosine(a, b):.3f}    (same direction, magnitude differs)")
print(f"cosine(a, c) = {cosine(a, c):.3f}    (opposite)")
print(f"euclid(a, b) = {euclid(a, b):.3f}    (different — magnitude matters)")
print(f"euclid(a, c) = {euclid(a, c):.3f}    (very different)")

cosine(a, b) = 1.000    (same direction, magnitude differs)
cosine(a, c) = -1.000    (opposite)
euclid(a, b) = 3.742    (different — magnitude matters)
euclid(a, c) = 7.483    (very different)

Look at b: it is just a doubled — same direction, twice the length. Cosine calls it a perfect 1.000 because it only reads the angle, while Euclidean reports 3.742 because the magnitudes differ. That difference is the whole reason cosine, not distance, is the default for comparing meaning.

The takeaways:

• Cosine is the default for semantic search. Most embedding APIs return unit-length vectors so cosine ≈ dot product.
• Euclidean is sensitive to magnitude — fine if your vectors are normalized (for unit vectors, smaller L2 distance corresponds to higher cosine similarity), but otherwise can mislead when vectors differ in length.
• Dot product is the fastest. Use it when vectors are already normalized; otherwise use cosine.

Matryoshka embeddings — truncate for cheaper search

Why normalized embeddings are the default

Major embedding providers (OpenAI, Cohere, Voyage) return unit-length vectors — ||v|| = 1 for every embedding. That makes cosine similarity reduce to a single dot product:

cos_sim(a, b) = dot(a, b)         # when ||a|| = ||b|| = 1

Faster, simpler, no division. Vector databases (next lesson) lean on this: they index normalized vectors and use dot product internally for speed.

Clustering — embeddings group by topic

The geometric setup means that similar texts cluster in the embedding space. We can use clustering algorithms (k-means, DBSCAN) to discover those groups automatically.

# Similar texts cluster. We mock an embedder with two meaning axes — a
# "tech" score and a "fitness" score — so the grouping falls out from
# plain distance, with no randomness and no library.
TECH = {"python", "code", "ml", "model", "models", "data", "tooling"}
FIT  = {"run", "running", "exercise", "fitness", "morning", "health"}

def embed(s):
    words = s.lower().replace(".", "").split()
    tech = sum(w in TECH for w in words)
    fit  = sum(w in FIT  for w in words)
    return (tech, fit)            # a 2D meaning vector: (tech-ness, fitness-ness)

sentences = [
    "Python is great for ML model training.",
    "We use code to build models.",
    "Data scientists ship ML models.",
    "I went for a run this morning.",
    "Exercise and fitness improve health.",
    "A daily run is great fitness.",
]

for s in sentences:
    tech, fit = embed(s)
    label = "tech" if tech >= fit else "fitness"
    print(f"({tech}, {fit})  {label:7s}  {s}")

(3, 0)  tech     Python is great for ML model training.
(2, 0)  tech     We use code to build models.
(3, 0)  tech     Data scientists ship ML models.
(0, 2)  fitness  I went for a run this morning.
(0, 3)  fitness  Exercise and fitness improve health.
(0, 2)  fitness  A daily run is great fitness.

The vectors land in two corners of the plane — high-tech/zero-fitness and zero-tech/high-fitness — so a clustering algorithm (k-means, DBSCAN) would separate them without ever being told what the topics are. Real embedders do the same thing in hundreds of dimensions, learned rather than hand-coded. That’s the magic of clustering on embeddings.

Visualizing — PCA and t-SNE

Embeddings live in hundreds of dimensions. To plot them on a screen, you need to reduce to 2D or 3D. Two tools:

• PCA (Principal Component Analysis) — fast, linear, preserves global structure. Use for a quick look.
• t-SNE — slower, non-linear, much better at preserving local clustering. Use when you actually want to see clusters.

# To plot hundreds of dimensions you reduce to 2. PCA finds the two axes
# that spread the data most; here we use two obvious ones — the tech
# features (first 5 dims) and the fitness features (last 3) — to show what
# the reduced plot looks like.
points = {
    "tech1": (1, 1, 1, 1, 0, 0, 0, 0),
    "tech2": (0, 1, 0, 1, 1, 0, 0, 0),
    "tech3": (1, 0, 0, 1, 1, 0, 0, 0),
    "fit1":  (0, 0, 0, 0, 0, 1, 1, 1),
    "fit2":  (0, 0, 0, 0, 0, 0, 1, 1),
    "fit3":  (0, 0, 0, 0, 0, 1, 1, 0),
}

def reduce_2d(v):
    return sum(v[:5]), sum(v[5:])      # (tech axis, fitness axis)

for label, v in points.items():
    x, y = reduce_2d(v)
    print(f"  {label:6s} -> (tech={x}, fit={y})")

  tech1  -> (tech=4, fit=0)
  tech2  -> (tech=3, fit=0)
  tech3  -> (tech=3, fit=0)
  fit1   -> (tech=0, fit=3)
  fit2   -> (tech=0, fit=2)
  fit3   -> (tech=0, fit=2)

Plot those six points and you see two obvious clumps — tech along one axis, fitness along the other. PCA and t-SNE find such spread-maximising axes automatically, so you don’t have to know them in advance. This is the diagnostic plot every embedding-debugging session starts with: “do my categories actually separate in vector space?” If they don’t, the embedding model isn’t capturing the distinction you want.

When the geometry lies

A few important caveats:

• Short texts embed poorly. “Yes” and “No” might end up close to each other — they’re both short and similar in part-of-speech. Don’t embed single tokens.
• Domain-specific jargon can fool a general-purpose embedder. A model trained on the open web may not distinguish “long position” from “short position” in finance. Fine-tune on domain data if quality matters.
• High cosine doesn’t mean “true.” Embeddings capture semantic similarity, not factual entailment. Two sentences saying contradictory things can have very similar embeddings.

In one breath

• An embedding is a point in high-dimensional space; near in space means near in meaning.
• Compare two vectors with cosine (angle, ignores magnitude — the default), Euclidean (straight-line distance), or dot product (fastest, equals cosine when vectors are unit-length).
• APIs return normalized vectors, so cosine reduces to a dot product — which is why vector DBs default to cosine/dot.
• Similar texts cluster, and reducing to 2D (PCA/t-SNE) lets you see whether your categories separate.
• Embeddings model similarity, not truth — two contradictory sentences on one topic embed alike; judging which is correct is the LLM’s job.

Quick check

Next

You’ve seen embeddings as geometric points and clustered a few by hand. The next lesson is how production systems actually store and query billions of these — vector databases.