Data augmentation

A small animal shelter gives you 1,200 labelled photos: cats and dogs. You hold out 200 images for validation and testing, then train a vision model on the other 1,000.

The model reaches 99% accuracy on the training photos. On new shelter photos, it reaches 78%.

Look at the training images and the answer is obvious. Many cats were photographed indoors. Many dogs were photographed outside. Several photos have the shelter logo in the corner. The model has found shortcuts. It has learned some animal features, but it has also learned “green grass means dog” and “this corner pattern means cat”.

You could collect another 100,000 labelled photos. That would help, if you have the money, time, and a second planet on which to store them.

There is a cheaper lever. Show the model plausible variations of the photos you already have: a cat shifted left, cropped slightly, made darker, or viewed as a horizontal mirror image. If the answer is still “cat”, the model gets practice ignoring those irrelevant changes.

That family of techniques is data augmentation: creating altered training examples from existing ones, with the target kept or changed according to a known rule.

The central idea: encode an invariance

An invariance is a change in the input that should not change the answer. For the shelter classifier:

• Moving the cat 20 pixels left should not change “cat”.
• A small brightness change should not change “cat”.
• A horizontal flip usually should not change “cat”.
• Replacing the cat with a dog absolutely should change the answer.

For ordinary classification augmentation, you believe the target stays the same after a valid transformation:

y(T(x)) = y(x)

Here, x is an image, y is its label, and T is a transformation such as a crop. This is a claim about the world: these two different-looking inputs should receive the same answer.

During training, the model sees original and randomly transformed versions. The loss is averaged over plausible transformations:

average loss = E_T[loss(model(T(x)), y)]

Without augmentation, the model can reduce training loss by memorising accidental details in particular pixels. With augmentation, those details move, disappear, or change. A shortcut that works on the original image becomes unreliable, while features that survive the transformation become more useful.

That is augmentation’s regularizing mechanism. It changes which explanations remain profitable for the model to learn. This is related to regularization, but more specific: augmentation enforces believable relationships between inputs and targets.

Not every augmentation keeps the target identical. Mixup and CutMix combine inputs and derive a new soft target. Detection and segmentation use equivariance: annotations change predictably with the input. Move an image 20 pixels right and its boxes and mask must move 20 pixels too.

The useful question is not “What augmentations are popular?” It is “Which changes should this task ignore, and how should the target change when they do not leave it alone?”

Basic image transformations

Image augmentations change either pixel location or appearance.

Geometric transformations include:

• Crop, which prevents reliance on fixed position but can remove the object.
• Resize, which helps with changing scale but can erase small details.
• Translation, which shifts the image.
• Rotation, perspective, and affine transforms, which simulate camera variation.
• Flip, which is valid only when the mirrored world has the same label.

Photometric transformations include brightness, contrast, colour, saturation, blur, sensor noise, shadows, and compression artefacts. They help when cameras, lighting, or distance vary, but hurt when colour or brightness carries the label. A “ripe” classifier should not aggressively destroy colour.

Here is a conservative pipeline for the shelter classifier. It assumes an ImageNet-pretrained model, so the final normalization uses the statistics that model expects. Normalization is not augmentation.

from torchvision import datasets, transforms

train_transform = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.8, 1.0)),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.ColorJitter(
        brightness=0.2,
        contrast=0.2,
        saturation=0.2,
        hue=0.05,
    ),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],
    ),
])

eval_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],
    ),
])

train_data = datasets.ImageFolder(
    "shelter_data/train",
    transform=train_transform,
)

validation_data = datasets.ImageFolder(
    "shelter_data/validation",
    transform=eval_transform,
)

Random settings are chosen when an item is fetched, so one source photo produces many views across epochs. Validation uses a deterministic transform so it measures performance on a stable, representative view.

For detection, bounding boxes must move with the image. For segmentation, the mask receives the same geometric transform, but not photometric changes: changing brightness must not change its class IDs.

Split before augmenting

Suppose 100 original photos each produce 10 transformed versions. If you randomly split the resulting 1,000 images, a rotated copy of a cat may be in training while a brightened copy is in validation. The model only needs to recognise the source photo’s fingerprints.

With an 80/20 split, the chance that all 10 versions of one original land in validation is 0.2^10, about one in 9.8 million. Almost every original will have a sibling in training, so validation can look excellent while real-world accuracy remains ordinary.

The safe order is:

1. Split original sources into training, validation, and test groups.
2. Apply random augmentation only when reading training examples.
3. Use a deterministic evaluation transform for validation and test.
4. Keep related images together: the same patient, video, household, scene, or burst belongs in one split.

This is data leakage: near-duplicate inputs from evaluation have reached training.

Mixup and CutMix

Mixup combines two inputs and their labels in the same proportions. For a cat [1, 0] and dog [0, 1], choose lambda = 0.7:

mixed image = 0.7 × cat image + 0.3 × dog image

mixed label = 0.7 × [1, 0] + 0.3 × [0, 1] = [0.7, 0.3]

The target says the input was made from 70% cat and 30% dog. The loss is equivalently:

0.7 × loss(cat) + 0.3 × loss(dog)

A network trained only on sharp class boundaries can become overconfident just outside its training examples. Mixup fills in paths between examples and asks predictions to change smoothly along them. This often improves accuracy and calibration, the relationship between confidence and correctness.

Mixup can blur meaningful boundaries, however. It is a poor fit when linear pixel mixtures have no useful interpretation or create impossible objects.

CutMix cuts a rectangle from one image and pastes it into another. The label is mixed according to the fraction of image area from each source. If 65% of the final image is cat and 35% dog, the target is [0.65, 0.35].

CutMix preserves local patches rather than blending every pixel, but area is not always a good measure of semantic contribution. A small patch can determine the answer in a traffic-sign task. Both methods require soft-label losses; pasting an image while keeping the first label creates systematic noise.

Test-time augmentation

Test-time augmentation (TTA) applies several valid views at inference and combines predictions. If the original cat view gives p(cat) = 0.80 and its flip gives 0.70, averaging gives p(cat) = 0.75.

TTA can reduce sensitivity to crop or framing, but two views usually mean roughly two forward passes. If one image takes 40 milliseconds, two may approach 80 milliseconds. Keep TTA only when its metric improvement justifies the latency and cost. Do not tune many variants against the test set.

For segmentation, transform predictions back to the original coordinates before averaging. Detection boxes need an appropriate merge procedure rather than a blind average.

Domain-specific augmentation

The same principle applies beyond images, but the transformations must match the modality.

For speech recognition, SpecAugment operates on a time-frequency representation such as a log-mel spectrogram. Frequency masking and time masking hide parts of the signal while leaving enough context for the transcript. Background noise, reverberation, gain, and modest speed changes can simulate recording conditions. Pitch changes may be harmless for speech content but destructive for music classification.

Text augmentation is harder because small edits can change meaning. Deleting “not”, replacing “excellent”, or removing the answer-bearing phrase in a question-answering input can change the label. Back-translation or strong paraphrasing can help some intent or sentiment datasets, but may introduce semantic drift or bias.

Across modalities, alter nuisance variation while preserving the information that defines the target.

Choosing and diagnosing augmentations

Start with transformations you can defend from how data is collected. Add one family at a time and compare against a clean baseline. Stronger policies are useful only after the basic label-preserving assumptions are sound.

Augmentation helps most when data is limited and deployment variation is understood. It cannot fix missing classes, broken labels, or a train–production shift that the transforms do not represent.

Common symptoms point to specific mistakes:

• Suspiciously high validation: look for transformed siblings, adjacent video frames, the same patient, or repeated scenes across splits.
• Both training and validation accuracy fall: reduce magnitude or probability. Visualize augmented batches; if a human cannot identify the object, the model cannot either.
• Errors cluster by orientation, lighting, or text style: remove the invalid transform and recheck the invariance with domain experts.
• Detection or segmentation collapses: render transformed images with boxes or masks to find coordinate errors.
• TTA doubles latency with no score improvement: remove redundant views. Large disagreement may mean the transformation is invalid or the model is too weak for TTA to rescue.

The honest limit

Augmentation cannot create information that the dataset never contained.

If your training set has no examples of a rare disease, rotating healthy images does not teach the model what the disease looks like. If all cameras are indoors, brightness jitter does not reproduce outdoor shadows, motion blur, lens distortion, and background changes in the right combination. If a class depends on a tiny mark that your crop removes, the augmentation has thrown away information.

Augmentation also preserves source bias. Ten thousand altered photos of the same 100 households are still evidence from 100 households. A flipped, blurred, colour-shifted example is not an independent observation of a new population.

When failure comes from missing coverage, collect representative data, improve labels, use a suitable pretrained model, or redesign sampling. Augmentation is a powerful prior about variation, not a data-generation machine.

What to remember

• Augmentation can encode an invariance, transform annotations with inputs, or combine inputs with derived targets.
• The right transform removes shortcuts; the wrong one manufactures label noise.
• Mixup and CutMix combine inputs and labels with soft targets.
• Split original sources before augmentation; augment training views, not evaluation data.
• Augmentation cannot supply missing classes, populations, or information.

Quick check

0/3

Q1

It makes the image larger than the model input.The flipped shape may no longer represent the same digit or writing convention.It changes the number of classes.It prevents the optimizer from computing gradients.

Check answer

Q2

[1.0, 0.0][0.5, 0.5][0.7, 0.3][0.3, 0.7]

Check answer

Q3

Increase the augmentation magnitude until validation accuracy rises.Put scans from each patient into only one split, then augment the training split.Apply the same random augmentation to the test set.Use label smoothing before checking the split.

Check answer