Self-supervised learning

At 3 a.m., your team gets a request that sounds reasonable and feels expensive: build a model that detects bicycles in street-camera images.

You have 10 million images. You have 8,000 bicycle labels.

Training only on the labels wastes most of the visual information. Training on all 10 million images with random targets teaches the network nothing useful. The missing trick is to make the images provide their own targets.

That trick is self-supervised learning: training a model with targets constructed from the data itself, rather than targets supplied by a human.

The model is not told “this is a bicycle”. It is given a task such as:

• recognise two altered views of the same image;
• reconstruct image patches that were hidden;
• predict a missing word from its surrounding words.

If the task is designed well, solving it requires the network to notice structure that later tasks also need.

The pretext-task idea

A pretext task is an artificial training problem whose answer can be generated automatically from unlabelled data. It is not the final job; it forces the network to learn.

Take one street image, x, and create two altered versions:

• view A: crop, resize, and change the colour slightly;
• view B: crop elsewhere, blur it, and flip it horizontally.

The target is that both views came from the same original image.

The network learns an encoder, a function that turns raw input into a compact vector. That vector is a representation: a numerical summary intended to preserve useful information. A bicycle classifier can later use it instead of starting from pixels.

The key is to remove easy answers. If every image from Camera 7 has a blue border, the model can identify the camera rather than the bicycle. Random crops and colour changes can remove that cue, making shapes, parts, and spatial relationships more useful.

A good pretext task has two properties:

1. The target is available for free. No human annotation is needed.
2. The target cannot be solved by a cheap nuisance cue. The model must learn structure that survives the transformations.

Rotation prediction illustrates the danger. Predicting whether an image was rotated by 0, 90, 180, or 270 degrees may teach object orientation—or merely that sky belongs at the top. The first may transfer to bicycle detection; the second is a weaker shortcut.

The central design question is:

What information must the model understand to predict the target, and what unwanted information can it exploit instead?

Contrastive learning: agree with the right thing

Contrastive learning makes related examples close together and unrelated examples farther apart, usually using cosine similarity between normalised vectors.

SimCLR starts with a batch of N images and makes two random views of each, producing 2N views. For an anchor view, its altered twin is the positive. Views from other original images are negatives.

The encoder and a small projection head, used only during pretraining, produce vectors z. SimCLR computes the loss on the projection output and discards the head before downstream evaluation, allowing the encoder to retain information beyond the pretext objective.

For anchor i and positive j, InfoNCE assigns:

p(i -> j) = exp(sim(z_i, z_j) / tau) / sum over k not equal to i of exp(sim(z_i, z_k) / tau)

Here sim is usually cosine similarity, and tau is the temperature, which controls how sharply similarity differences affect the probability. The denominator includes the positive and every candidate except the anchor. The loss is the negative logarithm of this probability. SimCLR applies it in both directions.

A worked number

Suppose one anchor has these cosine similarities:

• positive: 0.8;
• negative 1: 0.3;
• negative 2: 0.1.

With temperature tau = 0.2, the logits are:

• positive: 0.8 / 0.2 = 4;
• negative 1: 0.3 / 0.2 = 1.5;
• negative 2: 0.1 / 0.2 = 0.5.

Exponentiating gives approximately:

• exp(4) = 54.6;
• exp(1.5) = 4.5;
• exp(0.5) = 1.6.

So:

54.6 / (54.6 + 4.5 + 1.6) = 0.899

The loss is:

-log(0.899) = 0.106

This is good but not perfect. The model can improve by increasing positive similarity or reducing negative similarity.

A smaller temperature, such as 0.05, makes the same differences much sharper; a slightly stronger negative can then take substantial probability. A larger temperature softens the distribution. Temperature sets the scale at which examples are separated.

Why large batches mattered

With N original images, each anchor has one positive and 2N - 2 negative candidates. A batch of 256 images gives 510 negatives; a batch of 4,096 gives 8,190.

More negatives make the task harder and provide a richer comparison set, which is why early SimCLR systems used large global batches across GPUs. But they consume memory and communication, and can contain false negatives: two photos of the same bicycle treated as unrelated. Pushing them apart damages the desired invariance.

A queue or memory bank, as in MoCo, supplies older embeddings when a batch cannot fit. This saves memory, but those negatives may be stale because the encoder has changed.

The collapse problem

A representation has collapsed when many inputs map to nearly the same vector. The loss may look calm even though a bicycle and a traffic light receive indistinguishable features. A frozen classifier then performs close to chance.

Negatives make collapse unattractive: if every vector is identical, every candidate has the same similarity, so the positive probability is about 1 / (2N - 1).

Some methods use no negatives.

BYOL has an online network and a target network. The online network encodes one view and a predictor matches the target network’s representation of another. The target branch uses stop-gradient, so gradients do not pass through it. An exponential moving average (EMA) of the online network updates the target.

DINO uses a similar teacher-student arrangement. The teacher is updated by momentum; its outputs are centred and sharpened before the student matches them. Multi-crop training gives the student local views while the teacher sees broader views.

Neither method has a universal mathematical guarantee against collapse. In BYOL, stop-gradient, EMA, the predictor, and normalisation create useful optimization asymmetry in practice. In DINO, centering counters collapse into one output dimension and sharpening counters uniform teacher targets. A badly tuned system can still produce low-variance, useless features.

Masked modelling: learn by filling the holes

A second family hides part of the input and asks the model to recover it.

In masked language modelling (MLM), a bidirectional network predicts hidden tokens from surrounding tokens. BERT-style training selects about 15 percent of positions; most selected tokens become mask tokens, some become random tokens, and some remain unchanged. This reduces dependence on a special mask symbol that will not appear during ordinary use.

A masked autoencoder (MAE) splits an image into patches, hides many, encodes only the visible ones, and uses a decoder to reconstruct the missing pixels. A common setting hides 75 percent.

Natural images are spatially redundant. Nearby sky patches reveal colour and texture, while visible edges constrain a missing object’s shape. Hiding many patches prevents the encoder from copying a nearly complete image and encourages broader structure.

Text is less redundant. Remove too many words from “the child put the glass on the table” and several completions may be equally plausible. A high mask ratio can leave too little context to learn syntax. The useful ratio depends on the modality’s redundancy, not simply on how much input is missing.

The reconstruction target also matters. MAE predicts pixels, providing a dense signal but potentially rewarding low-level texture. An encoder may reconstruct smooth sky without learning the category “bicycle”. Downstream evaluation must test whether it learned the information you need.

Choosing between objectives

A joint-embedding objective asks which inputs should have similar representations. Contrastive learning, BYOL, and DINO directly shape distances, making them natural for retrieval, clustering, and few-label classification.

A generative objective asks what content is likely from context. MLM, MAE, and autoregressive next-token prediction model missing, future, or fine-grained content. They better suit generation and reconstruction, though they may spend capacity on details a classifier does not need.

Neither is universally superior. Choose the objective according to what the downstream interface must preserve.

How to tell whether the representation works

A low pretraining loss only proves that the model solved the artificial task. It does not prove transfer.

The simplest test is a linear probe. Freeze the encoder and train only:

class scores = W * representation + b

If a small linear layer separates bicycle from non-bicycle images, the relevant information is already arranged simply. The test is intentionally weak: it asks whether the encoder did the hard work.

A second test is k-nearest neighbours (k-NN). Find the k training representations with highest cosine similarity and vote among their labels. With k = 5, four bicycle neighbours and one non-bicycle neighbour produce a bicycle prediction. No classifier is trained, so k-NN reveals local structure.

Use both:

• linear probing tests global separability;
• k-NN tests whether nearby examples share labels.

For example, 90 percent linear-probe accuracy but 62 percent 5-NN accuracy suggests global separability with mixed local neighbourhoods. Keep the encoder frozen, use identical splits across methods, report class balance, and test for leakage. A random split can let camera identity—or adjacent video frames—appear in both training and test sets.

A compact method guide

Method	Target	Strength	Main trap
Contrastive	Augmented positives and other examples	Retrieval, clustering, and classification features	False negatives, memory, and shortcut augmentations
BYOL or DINO	Slowly moving teacher target	No explicit negative queue	Collapse and low-variance bugs
MAE or MLM	Hidden pixels or tokens	Context learning from unlabelled data	Reconstruction may reward irrelevant detail
Autoregressive	Next token or element	Conditional modelling and generation	Capacity spent on frequent local details

The deciding axis is the information your downstream system must preserve: retrieval needs a useful distance, while generation needs a conditional model. A small labelled dataset may justify self-supervised pretraining followed by a linear probe or careful fine-tuning.

A production pattern

For the street-camera example:

1. Audit the corpus. Remove corrupt files and near-duplicates; record camera, location, and time because they can become shortcuts.
2. Choose task-aware transformations. A horizontal flip may be harmless for bicycle detection but harmful for text on road signs. A crop that removes the bicycle is not always a valid positive pair.
3. Pretrain while monitoring. Track loss, embedding variance, similarity distributions, and global-batch negatives when required.
4. Freeze and evaluate. Run linear probing and k-NN on a clean labelled validation set before fine-tuning.
5. Test shifts. Hold out a location, weather condition, or camera generation.

Common failures have recognisable symptoms:

• Collapse: feature standard deviation approaches zero and k-NN predicts one class. Check stop-gradient, target momentum, normalisation, predictor heads, and augmentation diversity.
• False negatives: retrieval pushes matching examples apart. Use metadata-aware or multi-positive losses, or a non-contrastive method.
• Too-strong augmentations: loss stays high and linear-probe accuracy is poor. Inspect paired views and reduce task-breaking crops.
• Shortcut features: random-split accuracy is high but a location-held-out split collapses. Remove identifying overlays and evaluate on the shift you fear.

The honest limitation is that self-supervision cannot create information absent from the data. Daytime road images cannot teach robust night-driving features, and a dataset without children cannot support reliable child-safety detection. “Same image under augmentation” is not necessarily “same meaning”, and pixel reconstruction is not understanding.

Self-supervised learning spends unlabelled data intelligently: it gives the network a reason to organise the world before labels arrive. Whether that organisation is useful still requires a downstream test.

What to remember

• Self-supervised learning creates targets from the input; it works only when shortcuts are harder than the structure you want.
• Contrastive methods bring augmented views together and compare them with other examples. Temperature controls the sharpness of those comparisons.
• Large batches provide negatives but cost memory and can create false negatives.
• BYOL and DINO avoid explicit negatives with teacher-student asymmetry and momentum updates; monitor for collapse.
• MAE and MLM hide content, but image redundancy supports much higher mask ratios than text.

Quick check

0/3

Q1

256510511512

Check answer

Q2

Images use floating-point values while text uses integers.Image patches have strong spatial redundancy, while removing too many text tokens can leave too little context.MAE does not use an encoder.Language models cannot be trained without human labels.

Check answer

Q3

Increase the classifier depth.Use a smaller batch without changing the data.Check whether the representation learned camera-specific shortcuts and evaluate the augmentations and data split.Replace cosine similarity with Euclidean distance everywhere.

Check answer