Norms & distances

The last lesson ended on a quiet warning: a vector’s length is not one fixed fact but a choice of ruler — the same arrow measured 5 units one way and 7 the other. Here is where that choice ripples to. Two of the most common questions in all of machine learning are “how big is this vector?” and “how far apart are these two?” — and both answers shift the moment you change the ruler. That ruler has a name, the norm, and choosing it is not a detail: it is the difference between ridge and lasso regression, between Euclidean and cosine nearest neighbours, between an embedding search that finds meaning and one that just finds long documents.

Norms: measuring size

A norm ||x|| is a way to assign a non-negative length to a vector. The important family is the Lp norms:

||x||_1   = |x_1| + |x_2| + ... + |x_n|              (L1 — "Manhattan" / taxicab)
||x||_2   = sqrt( x_1^2 + x_2^2 + ... + x_n^2 )       (L2 — "Euclidean", the usual length)
||x||_inf = max( |x_1|, |x_2|, ..., |x_n| )           (L-infinity — the largest component)

and in general ||x||_p = (Σ |x_i|^p)^(1/p). L2 is the straight-line length you know from geometry; L1 is how far you’d walk on a city grid; L∞ is just the biggest coordinate.

Distances: norms of a difference

Every norm gives a distance: the distance between u and v is the norm of their difference, ||u − v||. So L2 gives Euclidean distance, L1 gives Manhattan distance, L∞ gives Chebyshev distance. One more is special — cosine similarity — which ignores magnitude and measures the angle between vectors:

cos(u, v) = (u · v) / (||u||_2 · ||v||_2)

It is 1 when the vectors point the same way, 0 when orthogonal, −1 when opposite — and cosine distance is 1 − cos. This is the right tool when direction carries the meaning and length is noise — exactly the case for text embeddings.

import numpy as np
u = np.array([3.0, 4.0])
v = np.array([0.0, 5.0])

print("L1(u)  :", np.abs(u).sum())                 # 3 + 4
print("L2(u)  :", np.sqrt((u**2).sum()))           # sqrt(9+16)
print("Linf(u):", np.abs(u).max())                 # max(3,4)

d = u - v
print("euclidean(u,v):", round(float(np.sqrt((d**2).sum())), 3))
print("manhattan(u,v):", np.abs(d).sum())
print("cosine sim     :", round(float(u @ v / (np.sqrt(u@u) * np.sqrt(v@v))), 3))

L1(u)  : 7.0
L2(u)  : 5.0
Linf(u): 4.0
euclidean(u,v): 3.162
manhattan(u,v): 4.0
cosine sim     : 0.8

The same vector u = (3, 4) has length 7, 5, or 4 depending on the norm — and the distance to v is 3.16 (Euclidean) or 4 (Manhattan). The cosine similarity 0.8 says u and v point in fairly similar directions even though their magnitudes differ.

The unit ball explains L1’s sparsity

The clearest way to see the difference is the unit ball — the set of all vectors of norm 1. Its shape is what gives each norm its character:

Where this shows up in ML

• Regularization. Adding ||w||_2^2 to the loss (ridge) shrinks weights smoothly; adding ||w||_1 (lasso) drives some to exactly zero — feature selection for free, straight out of the diamond’s corners. The norm you penalize is the regularizer (see L1, L2, Elastic Net).
• Nearest neighbors and clustering. The distance metric defines who is “near”: Euclidean for dense continuous features, Manhattan when you want robustness to outliers, cosine when only direction matters (see kNN).
• Embeddings and retrieval. Semantic similarity between text/image vectors is almost always cosine — two embeddings mean the same thing when they point the same way, regardless of length (see embeddings).

In one breath

• A norm ||x|| measures a vector’s size; the Lp family covers L1 (sum of absolute values), L2 (Euclidean length), and L∞ (largest component).
• Every norm induces a distance via ||u − v|| — Euclidean, Manhattan, Chebyshev; cosine similarity is different, measuring the angle (direction), not magnitude.
• The unit ball’s shape is the intuition: L1 is a diamond with corners on the axes (→ sparse solutions), L2 a circle, L∞ a square.
• In ML the choice is load-bearing: L2 vs L1 regularization (ridge shrinks, lasso zeros), the distance metric in kNN/clustering, and cosine for embeddings.
• Match the metric to the geometry — standardize for Euclidean, Mahalanobis for correlated features, cosine for direction-only similarity.

Practice

A question to carry forward

We can now measure a vector two ways — its size, and its distance from another. But notice what we have not done: in every example the vectors sat perfectly still while we held a ruler to them. Nothing moved. Yet the entire point of a model is that data does not sit still — a sample gets rotated, stretched, projected into a new space, squeezed from a thousand dimensions down to ten — and each prediction is the end of a chain of such moves.

So here is the thread onward: what is the single object that performs one of those moves — that takes a vector in and hands a new vector back, turning or stretching or flattening the whole of space in one stroke? And why does it turn out that this “function that acts on vectors” is nothing more exotic than a grid of numbers — the very same matrix we have been storing our data in all along?