CLIP: contrastive vision-language
A ViT encodes images, a transformer encodes text — but in separate spaces you can't compare. CLIP trains both encoders together with a contrastive objective so images and their captions land in one shared embedding space. The payoff is zero-shot classification, and CLIP became the bridge that connects vision to language across modern multimodal AI.
Written and reviewed by Shreyash Prashu Last reviewed
What you'll learn
- The shared image-text embedding space, and why it's powerful
- Contrastive pretraining — matching the diagonal of a similarity matrix
- Zero-shot classification without any task-specific training
- Why CLIP is the bridge under text-to-image and vision-language models
Before you start
A ViT (a transformer that reads an image as a grid of patches) gives you an image embedding, which is a list of numbers that captures the image’s meaning. A text transformer gives you the same kind of list for a sentence. But the two live in separate spaces, so you can’t meaningfully compare a picture to a sentence. CLIP (Contrastive Language-Image Pretraining, 2021) trains the two encoders together so that an image and its caption land in one shared space. That single change unlocks zero-shot recognition (recognizing classes the model was never explicitly trained on). It also becomes the connective tissue of modern multimodal AI, meaning systems that handle several kinds of data at once.
Contrastive pretraining: match the diagonal
CLIP trains on ~400 million (image, caption) pairs scraped from the web. Each batch of N pairs
goes through an image encoder and a text encoder. You then compute the N×N matrix of cosine
similarities (a score from -1 to 1 for how closely two embeddings point the same way) between every
image and every caption. The contrastive objective says: make each image most similar to its
own caption, which sits on the diagonal, and dissimilar to the other N−1 captions:
import numpy as np
img_emb = np.array([[1.0, 0.0], [0.0, 1.0], [0.7, 0.7]]) # 3 image embeddings
txt_emb = np.array([[0.9, 0.1], [0.1, 0.9], [0.6, 0.8]]) # their 3 caption embeddings
def norm(M): return M / np.linalg.norm(M, axis=1, keepdims=True)
S = norm(img_emb) @ norm(txt_emb).T # cosine similarity matrix
print("similarity matrix (rows=images, cols=captions):")
print(np.round(S, 2))
print("predicted caption per image (argmax):", S.argmax(1), " <- the diagonal")similarity matrix (rows=images, cols=captions):
[[0.99 0.11 0.6 ]
[0.11 0.99 0.8 ]
[0.78 0.78 0.99]]
predicted caption per image (argmax): [0 1 2] <- the diagonalEach image’s highest-similarity caption is its own (the diagonal, [0 1 2]). Training pushes the
diagonal up and the off-diagonal down across hundreds of millions of pairs, and the result is a
shared space where related images and text are close — a picture of a dog and the words “a dog”
end up neighbors.
Zero-shot classification
The shared space gives CLIP its headline trick: classify images with no task-specific training. To recognize, say, dogs vs cats vs birds, you don’t fine-tune on labeled examples — you embed the label texts (“a photo of a dog”, “a photo of a cat”, “a photo of a bird”), embed the image, and pick the nearest text. Because images and text share the space, this works for any class you can describe in words, including ones never in a labeled training set. That generality — recognition driven by natural-language descriptions rather than a fixed label set — was the breakthrough.
In one breath
- A ViT and a text transformer embed into separate spaces; CLIP trains them together so images and captions share one space you can compare across.
- Contrastive pretraining on ~400M web (image, caption) pairs maximizes the diagonal of the
image-caption similarity matrix (match each image to its own caption) and minimizes the off-diagonal
(the demo: argmax per image is the diagonal
[0 1 2]). - The result is a shared space where related images and text are neighbors.
- This enables zero-shot classification: embed candidate label texts, embed the image, pick the nearest — recognizing any class you can describe in words, with no fine-tuning.
- CLIP is the multimodal bridge: its text encoder conditions text-to-image generators, its image encoder is the visual front-end of VLMs, and CLIP-score grades prompt adherence; web-scale weak supervision beat curated labels.
Quick check
Quick check
Next
CLIP pairs a ViT image encoder with a text encoder in a shared space; that space generalizes the embeddings idea across modalities, and its contrastive training mirrors sentence embeddings. Its text encoder conditions latent diffusion.
Practice this in an interview
All questionsA multimodal vision-language model turns image regions into visual tokens, maps them into the language model's hidden representation, and combines them with text through attention. CLIP is a contrastively trained dual encoder that aligns image and text embeddings; it can provide a vision backbone or support retrieval and zero-shot classification, but it is not by itself a generative visual question-answering model.
A transformer needs a positional signal because self-attention compares and mixes token content without knowing sequence order. Without positional information, its shared layers are permutation-equivariant, so it cannot reliably model order-dependent relationships such as who did what to whom.
Encoder-only transformers read the whole input bidirectionally and produce representations, decoder-only transformers use causal attention to predict the next token, and encoder-decoder transformers encode a source sequence before a causal decoder generates a target sequence with cross-attention. The choice follows the task: understanding, open-ended continuation, or sequence-to-sequence transformation.
An embedding is a dense, learned vector representation of a discrete or high-dimensional object — a word, image, user, product — in a continuous low-dimensional space. Proximity in embedding space reflects semantic or behavioural similarity, making embeddings a universal interface between raw data and neural networks.