Matrices as data and as transformations

The last lesson promised a single object that moves a vector — rotates it, stretches it, flattens a whole space in one stroke — and hinted it would turn out to be the very grid of numbers we already use to hold data. That is the quiet double life of a matrix. Sometimes it is a table of data: rows are samples, columns are features. Sometimes it is a transformation: feed it a vector and it hands back a new one. Both readings are correct, and ML papers and code slide between them without ever announcing the switch. Learn to feel which hat a matrix is wearing in a given line and a great deal of notation stops being ambiguous.

Reading 1: matrix as data

When you see X in y = Xw + b, the matrix is wearing its data hat:

n rows, d columns: this is the design matrix. Everything in scikit-learn — X_train, X_test, the X you pass to .fit(X, y) — is shaped like this. Reductions along an axis answer the everyday questions: average a column for a per-feature mean, sum a row for a per-sample total.

import numpy as np

# A tiny dataset: 5 samples, 3 features
X = np.array([
    [3.2, 0.8, 1.0],
    [2.9, 1.1, 0.7],
    [3.4, 0.9, 1.2],
    [3.1, 1.0, 0.9],
    [3.0, 0.7, 1.1],
])

print("Shape:", X.shape)
print("Column means:", X.mean(axis=0))   # per-feature average
print("Row sums:",     X.sum(axis=1))     # per-sample total

Shape: (5, 3)
Column means: [3.12 0.9  0.98]
Row sums: [5.  4.7 5.5 5.  4.8]

The transpose X.T flips rows and columns — a move you make constantly, whenever an equation wants the features in rows rather than columns.

Reading 2: matrix as transformation

Now the other hat. A (2, 2) matrix is a function: hand it a 2-D vector and it returns a 2-D vector. Different matrices do different things to space — and the cleanest way to see what a matrix does is to watch what it does to a humble unit square:

import numpy as np

# A unit square: 4 corners stored as columns
square = np.array([
    [0, 1, 1, 0],
    [0, 0, 1, 1],
])

S = np.array([[2, 0], [0, 2]])     # scale
print("Scaled:");  print(S @ square)

R = np.array([[0, -1], [1, 0]])    # rotate 90 degrees
print("Rotated:"); print(R @ square)

H = np.array([[1, 1], [0, 1]])     # shear
print("Sheared:"); print(H @ square)

Scaled:
[[0 2 2 0]
 [0 0 2 2]]
Rotated:
[[ 0  0 -1 -1]
 [ 0  1  1  0]]
Sheared:
[[0 1 2 1]
 [0 0 1 1]]

Here is the insight that makes all of this tactile: a matrix sends î ([1, 0]) and ĵ ([0, 1]) — the two unit basis vectors — to its first and second columns, and every other point of space is dragged along linearly. Read a matrix’s columns and you already know where the whole grid goes. Drag the basis-vector tips below and watch the grid warp; the determinant readout is the factor by which areas scale, and a negative value means the orientation flipped.

This is exactly what a neural network’s linear layer is — a transform applied to a d_in-dimensional input to produce a d_out-dimensional output. Layers do not “look at” data so much as warp the space the data lives in until the classes pull apart.

The identity matrix: do nothing

Among all transforms there is one that changes nothing at all — the identity I, with 1s on the diagonal and 0s everywhere else:

For any vector x, I @ x == x; for any matrix A, I @ A == A. It is the matrix world’s version of multiplying by 1.

import numpy as np

I = np.eye(3)
print(I)

x = np.array([5.0, -2.0, 7.0])
print("I @ x:", I @ x)             # unchanged

[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
I @ x: [ 5. -2.  7.]

np.eye(n) is your friend: the identity turns up as the “do nothing” starting point of many ML formulas — including, as you are about to see, the small nudge that rescues a broken inverse.

The inverse: undo a transform

If A is a transformation, its inverse A⁻¹ is the transformation that undoes it: A⁻¹ @ A == I. This is the engine of closed-form linear regression,

w = (Xᵀ X)⁻¹ Xᵀ y

where the (Xᵀ X)⁻¹ term inverts a matrix and, when that inverse exists, hands you the optimal weights in a single calculation — no gradient descent required.

import numpy as np

# A nice invertible 2x2
A = np.array([
    [2.0, 1.0],
    [1.0, 3.0],
])

A_inv = np.linalg.inv(A)
print("Inverse:")
print(A_inv.round(3))

# Sanity: A @ A_inv should be the identity
print("A @ A_inv =")
print((A @ A_inv).round(3))

Inverse:
[[ 0.6 -0.2]
 [-0.2  0.4]]
A @ A_inv =
[[ 1.  0.]
 [-0.  1.]]

(The -0. is floating point’s harmless way of writing zero — the result is the identity.)

When inverses break (singular matrices)

A matrix is singular when its columns are linearly dependent — one column is a combination of the others — so the transform collapses some direction down to nothing, and nothing can un-collapse it.

import numpy as np

# col_2 = 2 * col_1 — perfectly correlated features
A = np.array([
    [1.0, 2.0],
    [3.0, 6.0],
])

try:
    np.linalg.inv(A)
except np.linalg.LinAlgError as e:
    print("Failed:", e)

print("Determinant:", np.linalg.det(A))   # essentially 0 -> singular

Failed: Singular matrix
Determinant: -3.330669073875464e-16

This is precisely what happens when two features are perfectly correlated: linear regression cannot choose weights, because infinitely many combinations fit equally well. Worse, with floating point an almost-singular matrix (look at that determinant: -3.3e-16, a hair off zero) gives wildly unstable answers. The standard fix is regularization — solve (XᵀX + λI)⁻¹ Xᵀy instead. Adding λI nudges the matrix back to invertible, and that nudge is exactly ridge regression.

The determinant: volume scale

The determinant reports how much a transform scales volumes (areas, in 2-D):

• det = 2 → doubles areas
• det = 1 → preserves area (rotations are like this)
• det = -1 → preserves area but flips orientation (a mirror)
• det = 0 → flattens space to a lower dimension → singular, no inverse

import numpy as np

scale_2x  = np.array([[2, 0], [0, 2]])
rotation  = np.array([[0, -1], [1, 0]])
flatten   = np.array([[1, 2], [2, 4]])    # cols are colinear

for name, M in [("scale 2x", scale_2x), ("rotation", rotation), ("flatten", flatten)]:
    print(f"{name:10s}  det = {np.linalg.det(M):.3f}")

scale 2x    det = 4.000
rotation    det = 1.000
flatten     det = 0.000

You will rarely compute a determinant by hand, but you will read the word constantly — “singular,” “rank-deficient,” “ill-conditioned” are all shades of “determinant near zero, this is about to explode numerically.”

In one breath

A matrix wears two hats: a table of data (rows are samples, columns are features — the design matrix X) and a transformation (it sends the basis vectors î, ĵ to its columns and drags all of space along, which is exactly what a linear layer does). The identity does nothing (I @ x = x); the inverse A⁻¹ undoes a transform but only exists when the columns are independent — a singular matrix (zero determinant) collapses a direction and cannot be inverted, which is why correlated features break linear regression and why ridge’s + λI nudge fixes it; and the determinant measures how much the transform scales volume, with zero meaning “flattened, no way back.”

Practice

A question to carry forward

We have applied a single transform — one matrix, one warp of space. But a model is rarely one move; it is a stack of them, layer after layer, each warping the output of the last. So picture two transforms in a row: rotate with R, then scale with S. You could push the data through R and then through S — two passes — or you could ask a sharper question: is there a single matrix that does the work of both at once?

There is, and building it is the operation people most often get backwards. Here is the thread onward: when you compose two matrices — apply one transform, then another — what does the combined matrix actually compute entry by entry, why is “first R, then S” written S @ R and not R @ S, and why does that reversed order make matrix multiplication the strangest-looking rule in all of linear algebra until the moment it suddenly makes sense?