Vectors, dot products, and norms

The last lesson left the word vector doing two jobs at once. It called a vector a row of the dataset — a plain list of numbers — and then, in the same breath, spoke of two vectors “pointing the same way.” Lists do not point anywhere. Arrows do. The trick, and it is the whole trick, is that both descriptions are of the same object.

A vector is [3.0, -1.2, 0.8] — a list of d numbers. That is the bookkeeping answer, and it is correct. The answer that makes machine learning easier is that those same d numbers are an arrow in d-dimensional space: it has a direction it points and a length it reaches, it can be added to another arrow, stretched, flipped, and compared by the angle between them. Make the flip from “list” to “arrow” and the two questions this whole chapter keeps circling — how long is this vector? and what is the angle between these two? — turn from algebra into something you can see.

Two pictures, same object

The very same pair of numbers [3, 2] can be drawn two ways: as an arrow from the origin out to a tip, or as a single point sitting where that tip lands.

Reach for whichever picture fits the task. “Arrow” is the better lens for operations like addition; “point” is the better lens for picturing a dataset, where each sample is one dot in space. Drag the arrow below and watch its numbers, length, and angle move together — that linked motion is the idea.

Addition and scaling

Adding two vectors follows the parallelogram rule: to form a + b, set the tail of b at the tip of a and read off where you land. Scaling by a number just stretches or shrinks the arrow without turning it (a negative number flips it around).

import numpy as np

a = np.array([3.0, 1.0])
b = np.array([1.0, 2.0])

print("a + b   =", a + b)        # tip-to-tail
print("2 * a   =", 2 * a)        # twice as long, same direction
print("-a      =", -a)           # flipped
print("a - b   =", a - b)        # a + (-b)

a + b   = [4. 3.]
2 * a   = [6. 2.]
-a      = [-3. -1.]
a - b   = [ 2. -1.]

It is not a deep idea, but you will meet it constantly. A gradient update w_new = w_old - lr * grad is exactly “stand at w_old and step in a direction (-grad) for a chosen length (lr).” Averaging two embeddings is (a + b) / 2, the midpoint of the parallelogram. The arrow arithmetic is quietly doing the work.

The dot product (similarity in disguise)

The dot product multiplies two vectors position by position and sums — but it has a second face that is where its power lives:

a · b = a[0]·b[0] + a[1]·b[1] + … + a[d-1]·b[d-1]
      = ||a|| · ||b|| · cos(θ)

That second line is the magic: the dot product equals the two lengths multiplied together, times the cosine of the angle between the arrows. So the sign and size of one number report the geometry directly.

• Same direction (θ = 0) → cos = 1 → dot product large and positive
• Perpendicular (θ = 90°) → cos = 0 → dot product exactly 0
• Opposite (θ = 180°) → cos = −1 → dot product large and negative

import numpy as np

a = np.array([1.0, 0.0])
b = np.array([1.0, 0.0])    # same direction
c = np.array([0.0, 1.0])    # perpendicular
d = np.array([-1.0, 0.0])   # opposite

print("a · b:", a @ b)      # aligned
print("a · c:", a @ c)      # perpendicular
print("a · d:", a @ d)      # opposite

a · b: 1.0
a · c: 0.0
a · d: -1.0

There is a catch hiding in that first form, though: because the dot product multiplies by both lengths, a very long vector can score a big dot product just for being long, not for pointing the right way. To compare pure direction, divide the lengths back out — that is cosine similarity:

import numpy as np

def cosine(u, v):
    return (u @ v) / (np.linalg.norm(u) * np.linalg.norm(v))

# Two product embeddings (made-up)
shoes_red  = np.array([0.8, 0.1, 0.5, 0.2])
shoes_blue = np.array([0.7, 0.2, 0.6, 0.1])
shirt      = np.array([0.1, 0.9, 0.0, 0.7])

print("red vs blue shoes:", round(cosine(shoes_red, shoes_blue), 3))
print("red shoes vs shirt:", round(cosine(shoes_red, shirt), 3))

red vs blue shoes: 0.978
red shoes vs shirt: 0.279

Two shoes sit at 0.978 — almost the same direction; a shoe and a shirt at 0.279 — barely related. Cosine similarity always lands between −1 and 1 no matter how long the vectors are, which is exactly why recommendation systems and semantic search prefer it: the length of an embedding should not get a vote on what counts as similar.

Projection: shadows

The dot product answers one more question: the projection of one vector onto another — how much of a lies along the direction of b. The scalar projection onto b is a · b̂, where b̂ = b / ||b|| is the unit vector (length 1) pointing the way b points. Geometrically, it is the shadow a casts on the line through b.

That shadow is the move behind linear regression (find the closest point on a hyperplane), PCA (project the data onto its principal axes), and constrained gradient descent (project the gradient onto the surface you are allowed to move on). One quiet operation, three famous methods.

Norms: how long is a vector?

A norm — also called magnitude or length — answers “how big is this vector?” Two of them cover almost everything you will do:

• L2 norm (Euclidean): ||v||₂ = sqrt(v[0]² + v[1]² + …). The ordinary straight-line length.
• L1 norm (Manhattan): ||v||₁ = |v[0]| + |v[1]| + …. The sum of absolute values — distance walked on a city grid, never cutting the corner.

import numpy as np

v = np.array([3.0, -4.0])

print("L2 norm:", np.linalg.norm(v, ord=2))   # the 3-4-5 triangle
print("L1 norm:", np.linalg.norm(v, ord=1))
print("Manual L2:", np.sqrt(v @ v))            # same as the L2 norm

# Normalize to unit length: same direction, length 1
v_unit = v / np.linalg.norm(v)
print("Unit:", v_unit, "Length:", np.linalg.norm(v_unit))

L2 norm: 5.0
L1 norm: 7.0
Manual L2: 5.0
Unit: [ 0.6 -0.8] Length: 1.0

The same arrow is “5 long” by the straight-line ruler and “7 long” by the city-grid ruler — proof that length is a choice, not a fact. That choice shows up directly in regularization: L2 regularization (weight decay, ridge) adds λ · ||w||₂² to the loss and gently discourages large weights, while L1 regularization (lasso) adds λ · ||w||₁ and pushes small weights all the way to exactly zero, handing you sparsity. Same idea of “size,” two different rulers, two different behaviours.

In one breath

A vector is one object with two faces: a list of numbers and an arrow with a length and a direction. The dot product reads the angle between two arrows (and cosine similarity strips out length so only direction counts); projection is the shadow one arrow casts on another; and a norm measures length — but “length” depends on the ruler you pick, L2 (straight-line) or L1 (city-grid), a choice that quietly decides how a model penalises its weights.

Practice

A question to carry forward

We measured a vector’s length with the L2 norm, mentioned the L1 norm almost in passing — and then watched that “almost in passing” turn out to matter, because the same arrow was 5 units long one way and 7 the other. That was a hint with consequences. “Length” is not a single fixed quantity; it is whatever norm you choose, and the norm you choose silently sets how your model measures the distance between two points and which points it considers “close.”

So here is the thread onward: looked at squarely as distances rather than just lengths, what are the L1, L2, and other norms really doing — how do they disagree about which neighbours are nearest, and why does that disagreement quietly change what a k-nearest-neighbours search, a clustering, or a regulariser actually learns?