How do you train a deep learning model when you have very little labelled data?

How to think about it

The short answer

Suppose I have 500 labelled photographs of solar panels, and I need to classify each panel as cracked or intact. I would not train a large neural network from random initialisation. I would start with a pretrained backbone, train a small classification head, use only label-preserving augmentation, validate with a leakage-resistant split, and then use unlabelled panel images for self-supervised or semi-supervised learning if they exist.

The important caveat is that “very little” depends on the task. Five hundred images may be workable for a simple binary classifier. Five hundred examples are nowhere near enough for a medical segmentation model that must outline tiny lesions pixel by pixel.

Why transfer learning is the first move

A backbone is the part of a neural network that turns raw input into useful features. In an image model, early layers may detect edges and textures; later layers may detect shapes and object parts. A task head is the small final layer that converts those features into the prediction you need.

Transfer learning means starting with weights learned on a much larger, related dataset and adapting them to your task. A ResNet or vision transformer pretrained on natural images, BERT pretrained on text, or Whisper pretrained on speech already contains useful structure. Your 500 labels then teach the model what “crack” means, rather than forcing it to discover edges, shapes, and visual regularities from scratch.

That matters because a randomly initialised network has millions of adjustable weights and only 500 supervised examples to constrain them. It can memorise the training images easily. It has much less evidence for learning a rule that survives a new camera angle, a cloudy day, or a different panel.

I usually train in two stages. First, freeze the backbone and train only the head. This gives the small labelled dataset a low-capacity problem. Second, if the validation results suggest the pretrained features are not quite right for the domain, unfreeze the last one or two backbone blocks and fine-tune them with a learning rate perhaps ten times smaller than the head’s learning rate.

Freezing everything forever can underfit when the target domain is unusual. Unfreezing the whole network immediately can make 500 examples rewrite useful general features into memorised noise. The two-stage approach lets the data earn the right to change more of the model.

A concrete 500-image setup

Imagine the 500 panel photographs came from 180 physical panels. I would split by panel, not by photograph:

• 350 images for training
• 75 images for validation
• 75 images for the final test

The grouping is more important than the exact percentages. If photos of the same physical panel appear in both training and test, the model may recognise its background, mounting hardware, or existing scratch instead of learning the difference between cracked and intact. The test score can look excellent while the first new farm camera produces embarrassing predictions.

A minimal transfer-learning setup might look like this:

import torch
import torch.nn as nn

from torchvision.models import ResNet50_Weights, resnet50

model = resnet50(weights=ResNet50_Weights.DEFAULT)

for parameter in model.parameters():
    parameter.requires_grad = False

model.fc = nn.Linear(model.fc.in_features, 2)

optimizer = torch.optim.AdamW(
    model.fc.parameters(),
    lr=1e-3,
    weight_decay=1e-4,
)

The new final layer remains trainable because it is created after the backbone is frozen. I would first train this head with early stopping based on validation loss. If the head cannot separate the classes, or if the domain differs strongly from the pretraining images, I would unfreeze the final residual stage, create an optimizer that includes those newly trainable parameters, and use a smaller learning rate for them.

The learning rate values are starting points, not sacred numbers. With only 350 training images, I would compare settings using grouped cross-validation or several fixed seeds rather than trusting one lucky run.

Augmentation helps, but it does not create new evidence

An augmentation is a deliberately altered training example whose label should remain valid. For the panel task, reasonable transformations might include modest changes in brightness and contrast, a horizontal flip if left-right orientation is irrelevant, and a crop that still leaves enough of the panel visible.

A vertical flip may be invalid if the model needs to understand how cracks appear relative to gravity or panel hardware. An aggressive crop may remove the crack entirely. A colour transformation that turns a genuine dark crack into a bright artificial mark can teach the wrong rule.

The mechanism is not that 500 images magically become 5,000 independent observations. Augmentation teaches an invariance: “this prediction should remain the same when the lighting changes slightly.” That can reduce memorisation, but all transformed versions of one photograph are still closely related. Augmentation cannot supply missing examples of rain, snow, different panel manufacturers, or a new camera lens.

<Callout type=“warn”> Apply random augmentation only while constructing training batches. Keep validation and test inputs untouched apart from deterministic preprocessing such as resizing and normalisation. If a validation image is randomly altered, its score now mixes model quality with the luck of the transformation; if augmented copies cross the split boundary, the evaluation can leak information from training. </Callout>

For text, the same rule is stricter. Synonym replacement or back-translation can change the label, especially for sentiment, legal language, and medical advice. For audio, noise injection is useful only when the deployed system will actually encounter comparable noise. The transformation must reflect a real nuisance, not merely make the training set look busy.

Use unlabelled data when it matches the deployment world

Suppose I also have 10,000 unlabelled panel photographs from the same inspection system. I can use self-supervised learning, where the data provide an artificial learning target, before using the 500 human labels.

For example, a model can hide parts of an image and learn to predict them, or learn that two altered views of the same image should have similar representations while views from different images should be distinguishable. The model learns the camera, lighting, textures, and panel structure without anyone drawing a crack boundary. I would then fine-tune that representation on the 500 labelled examples.

This works best when the unlabelled data resemble production data. Ten million unrelated holiday photographs may be less useful than 10,000 images from the actual inspection cameras. Domain match determines whether the extra data teach useful structure or simply reinforce irrelevant features.

Semi-supervised learning combines labelled and unlabelled examples. One common approach is pseudo-labelling: use a model to assign provisional labels to unlabelled examples, then train on the examples judged sufficiently reliable. I would use confidence thresholds and consistency checks, and monitor the provisional labels by class. A model that is confidently wrong can multiply its own mistake; this is confirmation bias wearing a lab coat.

Regularisation and evaluation

Regularisation means adding constraints that make memorisation less attractive. I would use a small task head, weight decay, early stopping, and possibly dropout or label smoothing, tuning them rather than blindly stacking every available technique.

I would also inspect class balance. If only 30 of the 500 panels are cracked, accuracy is a poor primary metric: a model that always predicts “intact” gets 94 percent accuracy and is useless. Depending on the cost of missed cracks and false alarms, I might report recall, precision, the precision-recall curve, and performance at the operating threshold the inspection team can actually use.

A 75-image test set also makes impressive-looking numbers fragile. One error changes accuracy by about 1.33 percentage points. I would report uncertainty or the spread across grouped folds, keep the final test set untouched until model selection is complete, and check calibration. Calibration means that predictions claiming 80 percent probability should be correct roughly 80 percent of the time in that population.

The first failure mode I look for is training accuracy racing toward 100 percent while validation loss rises. That usually means memorisation, an over-capable head, weak augmentation, noisy labels, or a faulty split. I would inspect duplicate and near-duplicate images, verify labels, reduce the number of trainable layers, and compare learning curves.

The opposite failure is a strong validation score followed by poor production performance. That points toward leakage or domain shift, meaning that the real inputs differ from the training distribution. I would compare camera models, locations, lighting, panel manufacturers, and time periods. A random image split may estimate performance on more photos from the same farms, not performance on next year’s farms.

When I would not use a deep network

If the 500 examples are rows of structured measurements such as temperature, voltage, age, and manufacturer, I would establish baselines with regularised logistic regression and gradient-boosted trees. These models often win on small tabular datasets because they need fewer examples to estimate useful relationships.

I would also question whether more modelling is the right answer. Ten carefully selected labels from a new failure mode may be worth more than 1,000 redundant labels. Active learning makes the model choose the next examples for a human to label, often focusing on uncertain or diverse cases. Label audits, better collection conditions, and weak labels from existing inspection rules can beat another week of hyperparameter tuning.

The senior answer is therefore not “always use transfer learning and augmentation.” It is “reduce the amount of new information the model must learn, make every label trustworthy, evaluate on genuinely unseen cases, and choose a model whose capacity matches the evidence.”

What they’ll ask next

Why not train from scratch if the network is powerful?

Because power increases the number of ways the model can fit accidental details in a tiny dataset. Pretraining supplies a useful prior: edges, shapes, language structure, or acoustic patterns already learned from much more data. Training from scratch becomes more reasonable when the dataset is genuinely large, the domain is radically different from available pretrained models, or the pretraining data introduce unacceptable bias or licensing constraints.

How do you know whether an augmentation is valid?

Ask whether a human would keep the label after seeing the transformed example, and whether the transformation represents a variation expected in production. I would visualise augmented samples, test ablations with and without each transformation, and never let augmentation copies cross the train, validation, or test boundary. For the panel example, brightness changes may be valid; cropping away the only crack is not.

What if the pretrained model is badly mismatched?

Use unlabelled data from the target domain for self-supervised adaptation, fine-tune more of the backbone gradually, and reduce the learning rate so useful features are not destroyed. I would compare several pretrained sources if available. If the mismatch remains severe and the labels are scarce, I would collect targeted labels or use a simpler model rather than pretending that a familiar architecture has solved the data problem.

Say this in the interview: “With very little labelled data, I would start from a related pretrained model, train conservatively with label-preserving augmentation and regularisation, exploit same-domain unlabelled data, and make leakage-resistant evaluation and label quality part of the model design.”