Convolutional neural networks (CNNs)

Before transformers ate the world, convolutional neural networks were how machines learned to see — and they still are the default for most vision tasks that aren’t worth a giant model. The whole idea is one small trick, repeated. Let’s earn it from first principles.

Why not just use a dense layer?

Take a modest 224×224 colour image. Flatten it and feed it to a fully connected layer with 1,000 units. That first weight matrix alone is 224 × 224 × 3 × 1000 ≈ 150 million parameters — for one layer, on a small image. Worse, it learns nothing reusable: a cat in the top-left corner and the same cat in the bottom-right are, to a dense layer, two completely unrelated input patterns. It would have to re-learn “cat” in every position.

Two facts about images break the dense approach, and point straight at the fix:

• Locality. A pixel’s meaning comes from its neighbours, not from a pixel 200 rows away. Edges, corners, textures are local.
• Translation invariance. An edge is an edge wherever it appears. The feature detector should be the same everywhere.

Convolution, concretely

A kernel is a small matrix of weights. You place it over a patch of the image, multiply element-wise, and sum — one number. Slide it one step, repeat. The grid of results is a feature map: high where the patch matched the kernel, low where it didn’t.

Hand-designed filters make it concrete: an edge-detector lights up only the rim of a shape, a blur smears it, identity copies it — same input, wildly different feature maps depending on the kernel. The window slides over every patch, filling the map one dot product at a time:

The magic of CNNs is that you don’t hand-pick these numbers. The nine kernel weights are parameters — gradient descent discovers which filters are worth having. Early layers reliably learn edge and colour detectors (the network rediscovers Sobel on its own); deeper layers learn textures, then parts, then objects.

Build it from scratch

Here is a valid 2-D convolution in pure NumPy — no deep-learning library, just the multiply-and-sum. Run it, then change the kernel to the blur np.ones((3,3))/9 and watch the output smooth out.

import numpy as np

# A tiny 6x6 "image" with a bright square in the middle
img = np.zeros((6, 6))
img[2:4, 2:4] = 1.0

# A vertical-edge detector (Sobel-x)
kernel = np.array([[-1, 0, 1],
                   [-2, 0, 2],
                   [-1, 0, 1]])

def conv2d(x, k):
    kh, kw = k.shape
    H, W = x.shape
    out = np.zeros((H - kh + 1, W - kw + 1))
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            patch = x[i:i+kh, j:j+kw]      # the receptive field
            out[i, j] = np.sum(patch * k)  # one dot product
    return out

fmap = conv2d(img, kernel)
print("input shape :", img.shape)
print("output shape:", fmap.shape, "  (valid conv shrinks by kernel-1)")
print(np.round(fmap, 1))
# The +/- pair shows the left and right edges of the square.

input shape : (6, 6)
output shape: (4, 4)   (valid conv shrinks by kernel-1)
[[ 1.  1. -1. -1.]
 [ 3.  3. -3. -3.]
 [ 3.  3. -3. -3.]
 [ 1.  1. -1. -1.]]

Notice the output is smaller than the input (6×6 → 4×4). A 3×3 kernel can’t centre on the border pixels, so a “valid” convolution trims a kernel − 1 rim. Two knobs control this:

• Padding — add a border of zeros so the output keeps the input size (padding=1 for a 3×3 kernel → “same” convolution). Without it, a deep stack would shrink the image away to nothing.
• Stride — how far the kernel jumps each step. Stride 2 skips every other position, halving the output size — a cheap way to downsample.

Channels: stacking filters

One kernel finds one kind of feature. Real layers learn many in parallel — say 64 kernels — producing 64 feature maps stacked into a 64-channel output. The next layer’s kernels are then 3×3×64: they look across all incoming channels at once, combining “edge here + colour there” into richer features. A colour input image is just the first case of this: 3 channels (R, G, B) in, out_channels feature maps out.

Pooling: throw away position, keep the signal

After a convolution + activation (usually ReLU), CNNs often pool: take each small region of the feature map and keep only its maximum (max pooling) or average. A 2×2 max-pool halves height and width and says “an edge fired somewhere in this region — I don’t care exactly where.” That deliberate loss of precise position is what buys translation invariance, and it shrinks the feature maps so deeper layers stay cheap.

The hierarchy: why depth works

Stack the block — conv → ReLU → pool — a few times and something powerful emerges. Each pooling step means a single deep-layer pixel now summarises a large patch of the original image: its receptive field grows. So the network builds a hierarchy:

pixels → edges → textures → motifs → object parts → objects
  (layer 1)  (layer 2)   (layer 3)    (deeper)      (head)

Early layers see tiny patches and learn generic edges; late layers see most of the image and learn “this is a wheel, that is a face.” A final dense layer (or global-average-pool) turns the top feature maps into class scores. That’s a CNN — LeNet, AlexNet, VGG, and ResNet are all this block, stacked deeper with better training tricks.

In one breath

• A dense layer on an image is hopeless — ~150M parameters for one small image, and it re-learns each feature in every position.
• A convolution learns one tiny kernel and slides it everywhere (weight sharing), exploiting locality and translation invariance — a few thousand params that work at any resolution.
• Stride and padding control output size; many kernels stack into channels; pooling drops exact position to buy translation invariance and downsample.
• Stacking conv → ReLU → pool grows each unit’s receptive field, building a hierarchy: edges → textures → parts → objects.
• ResNet-style CNNs are still the pragmatic default for limited-data, low-latency vision; ViTs win at large data but need much more of it.

Check yourself

What to remember

• A convolution slides one small learned kernel across the whole input — weight sharing turns a 150M-parameter dense layer into a few thousand reusable weights that work at any resolution.
• Stride and padding control output size; channels stack many filters; pooling trades exact position for translation invariance.
• Stacking conv → ReLU → pool grows the receptive field, building a hierarchy from edges to objects — the reason depth matters in vision.