Vision Transformers (ViT)

CNNs dominated computer vision for a decade because convolution bakes in exactly the right assumptions for images — locality and translation equivariance. The Vision Transformer (ViT, 2020) threw that out and asked: what if you run a plain transformer — the language architecture — directly on an image? The one idea that makes it work is turning the image into a sequence of patches.

Patchify: an image as a sequence

A transformer consumes a sequence of tokens. An image is a grid of pixels. ViT bridges them by splitting the image into fixed patches (e.g. 16×16 pixels), flattening each patch into a vector, and treating each as a token:

import numpy as np

img = np.arange(16).reshape(4, 4)        # a toy 4x4 grayscale image
P = 2                                    # 2x2 patches

patches = [img[i:i+P, j:j+P].flatten() for i in range(0, 4, P) for j in range(0, 4, P)]
print(f"{len(patches)} patch tokens of dim {patches[0].size}:")
for k, p in enumerate(patches):
    print(f"  patch {k}: {p}")

4 patch tokens of dim 4:
  patch 0: [0 1 4 5]
  patch 1: [2 3 6 7]
  patch 2: [ 8  9 12 13]
  patch 3: [10 11 14 15]

A 4×4 image becomes 4 tokens — a sequence. A real 224×224 image with 16×16 patches becomes 196 tokens. From there, ViT is a standard transformer encoder: linearly project each patch to the model dimension, prepend a [CLS] token (whose final state classifies the image), add positional embeddings (attention is permutation-invariant, so without them the model wouldn’t know where each patch sits), and run self-attention over the patch sequence.

CNN inductive bias vs ViT data hunger

CNNs come with built-in assumptions: a convolution only looks locally and slides the same filter everywhere (translation equivariance). Those biases are correct for images, so CNNs learn well from modest data. ViT has almost none of them — patches start unrelated and the model must learn spatial structure from scratch. The consequence: ViT underperforms CNNs on small datasets but, given large-scale pretraining (hundreds of millions of images), matches or beats them — and its global self-attention relates any two patches in one layer, where a CNN needs depth to build a large receptive field. More flexible, but hungrier for data.

In one breath

• The Vision Transformer runs a plain transformer on images by splitting the image into fixed patches and treating each flattened patch as a token (the demo: a 4×4 image → 4 tokens; 224×224 with 16×16 patches → 196).
• The pipeline: linearly project patches, prepend [CLS] (classifies), add positional embeddings (attention is order-agnostic), run self-attention.
• CNNs bake in locality + translation equivariance, so they’re data-efficient; ViT has little inductive bias, so it’s data-hungry but, with large-scale pretraining, matches/beats CNNs and relates any two patches via global attention.
• ViT’s real impact is unification — one transformer for text/images/audio/video by changing the tokenization.
• That’s what enabled multimodal models: a ViT image encoder shares architecture/space with a text transformer (the basis of CLIP and VLMs).

Quick check

Next

ViT is the image encoder that, paired with a text encoder in a shared space, becomes CLIP. It applies self-attention from the transformer to images, the alternative to CNNs.