CLIP: contrastive vision-language

A ViT gives you an image embedding; a text transformer gives you a text embedding. But they live in separate spaces — you can’t meaningfully compare a picture to a sentence. CLIP (Contrastive Language-Image Pretraining, 2021) trains the two encoders together so that an image and its caption land in one shared space. That single change unlocks zero-shot recognition and becomes the connective tissue of modern multimodal AI.

Contrastive pretraining: match the diagonal

CLIP trains on ~400 million (image, caption) pairs scraped from the web. Each batch of N pairs goes through an image encoder and a text encoder; you compute the N×N matrix of cosine similarities between every image and every caption. The contrastive objective: make each image most similar to its own caption — the diagonal — and dissimilar to the other N−1 captions:

import numpy as np

img_emb = np.array([[1.0, 0.0], [0.0, 1.0], [0.7, 0.7]])     # 3 image embeddings
txt_emb = np.array([[0.9, 0.1], [0.1, 0.9], [0.6, 0.8]])     # their 3 caption embeddings

def norm(M): return M / np.linalg.norm(M, axis=1, keepdims=True)
S = norm(img_emb) @ norm(txt_emb).T                          # cosine similarity matrix

print("similarity matrix (rows=images, cols=captions):")
print(np.round(S, 2))
print("predicted caption per image (argmax):", S.argmax(1), " <- the diagonal [0 1 2]")

similarity matrix (rows=images, cols=captions):
[[0.99 0.11 0.6 ]
 [0.11 0.99 0.8 ]
 [0.78 0.78 0.99]]
predicted caption per image (argmax): [0 1 2]  <- the diagonal

Each image’s highest-similarity caption is its own (the diagonal, [0 1 2]). Training pushes the diagonal up and the off-diagonal down across hundreds of millions of pairs, and the result is a shared space where related images and text are close — a picture of a dog and the words “a dog” end up neighbors.

Zero-shot classification

The shared space gives CLIP its headline trick: classify images with no task-specific training. To recognize, say, dogs vs cats vs birds, you don’t fine-tune on labeled examples — you embed the label texts (“a photo of a dog”, “a photo of a cat”, “a photo of a bird”), embed the image, and pick the nearest text. Because images and text share the space, this works for any class you can describe in words, including ones never in a labeled training set. That generality — recognition driven by natural-language descriptions rather than a fixed label set — was the breakthrough.

In one breath

• A ViT and a text transformer embed into separate spaces; CLIP trains them together so images and captions share one space you can compare across.
• Contrastive pretraining on ~400M web (image, caption) pairs maximizes the diagonal of the image-caption similarity matrix (match each image to its own caption) and minimizes the off-diagonal (the demo: argmax per image is the diagonal [0 1 2]).
• The result is a shared space where related images and text are neighbors.
• This enables zero-shot classification: embed candidate label texts, embed the image, pick the nearest — recognizing any class you can describe in words, with no fine-tuning.
• CLIP is the multimodal bridge: its text encoder conditions text-to-image generators, its image encoder is the visual front-end of VLMs, and CLIP-score grades prompt adherence; web-scale weak supervision beat curated labels.

Quick check

Next

CLIP pairs a ViT image encoder with a text encoder in a shared space; that space generalizes the embeddings idea across modalities, and its contrastive training mirrors sentence embeddings. Its text encoder conditions latent diffusion.