Transfer learning and fine-tuning

At 3 a.m., a factory manager sends you 500 photographs of solder joints. Each image is labelled good or cracked. You need a classifier by Friday.

A neural network built from random weights has to learn everything: what an edge is, how colours change across a surface, which shapes form a component, and which component looks cracked. With 400 training images, it will happily memorise the photographs. It will not become a reliable inspector.

A large image model has already seen roughly 1.2 million natural images. It has learned useful visual machinery long before you arrived. The sensible move is to keep that machinery and teach only what your factory cares about.

That move is transfer learning: starting with a model trained on one task and reusing its learned representations on another. Fine-tuning is the narrower step of continuing training on the new task, changing some or all of the pretrained weights.

The rest of this lesson uses the solder-joint classifier as its running example.

What a pretrained network already knows

A neural network is a chain of transformations. Each layer turns its input into a new representation: numbers arranged to preserve information useful to the next layer.

In an image classifier, early filters often respond to local patterns such as edges, colour transitions, corners, and textures. Later layers combine them into larger structures—circles, holes, pins, or the geometry of a solder joint.

The final layers are shaped directly by the original labels. An ImageNet model’s last layers separate its 1,000 object categories, so they are less likely to identify a hairline crack.

Why does this progression appear? A vertical-edge detector can help recognise a wheel, a bird’s beak, or a circuit-board trace. Gradients from many categories reward such reusable features. A later unit that combines edges into a “wheel-like circle” helps fewer tasks, so it becomes more specific to the original problem.

This is a tendency, not a law. Infrared images, radar, medical scans, and ordinary photographs have different low-level statistics. Even so, early layers are usually the safest to transfer.

A typical network therefore looks conceptually like this:

A backbone is the reusable feature extractor, usually everything before the final classifier. A head is the task-specific output layer. For solder joints, remove the ImageNet head and attach one with two outputs: good and cracked.

“Freeze the backbone” means stop updating its parameters. It still performs its forward pass and turns pixels into features.

The decision: data size crossed with domain similarity

Choose how much to change by asking:

1. How many labelled examples do you have?
2. How similar is the new input domain to pretraining?

Domain similarity includes appearance, resolution, colour distribution, textures, and which visual patterns carry the signal. Two datasets can both contain three-channel images while having very different domains.

Dataset	Similar domain	Different domain
Small	Start with a linear probe; cautiously unfreeze the last block if validation supports it.	Freeze early layers and adapt cautiously; consider domain-specific pretraining or more labels.
Large	Full fine-tuning is usually sensible.	Compare full fine-tuning, domain-adaptive pretraining, and training from scratch if data and compute allow.

There is no magic boundary at 1,000 or 100,000 examples. A dataset of near-duplicate frames contains less information than a smaller, varied dataset. For 500 solder images, start with a frozen backbone and a new head. This tests whether the pretrained features contain useful signal before you risk changing them.

Three ways to adapt the model

1. Linear probing

Linear probing freezes the backbone and trains only a linear classifier on its output features.

A ResNet-18 produces a 512-number feature vector after pooling. A two-class head contains 512 weights for each class and 2 bias values: 1,026 trainable numbers, compared with about 11.7 million parameters in the whole network. With 400 training images, that is much less prone to memorisation.

The head learns a boundary between fixed feature vectors. If cracked joints already occupy a different region because of their shapes or textures, this can work well. With AdamW, 1e-3 is a reasonable starting learning rate for the new head.

Linear probing is also a diagnostic. If it cannot beat a majority-class baseline, check the representation, preprocessing, and labels before unfreezing everything.

2. Full fine-tuning

Full fine-tuning updates the backbone as well as the new head. It gives the model freedom to reshape features for the new domain, but it can ruin a useful model with 500 examples.

The random head initially produces unhelpful errors. Those errors send large, noisy gradients through the backbone, which can destroy broadly useful features before the task boundary stabilises. Training loss may fall while validation accuracy collapses.

Use a lower learning rate for pretrained layers than for the head. For example:

• new head: 1e-3
• last backbone block: 1e-4
• early layers: 1e-5

This layer-wise or discriminative learning rate is causal: random parameters need large corrections, while established parameters need gentle nudges.

3. Progressive unfreezing

Progressive unfreezing starts with the head, then allows later backbone blocks, and finally earlier blocks, to update:

• epochs 1–5: head only
• epochs 6–10: last block
• after epoch 10: more blocks only if validation improves

The order follows specialisation. The head needs the most change; early layers are usually most reusable. Stop at the first level that helps.

Catastrophic forgetting

Catastrophic forgetting is the rapid loss of performance on an original or retained task when continued training overwrites useful pretrained representations.

A drop on the new task’s validation set is not, by itself, evidence of forgetting. For example, if solder validation accuracy rises to 88 percent and then falls to 61 percent while training accuracy reaches 100 percent, that shows overfitting or transfer degradation. To test forgetting, measure an original-task or retained-task benchmark before and after fine-tuning.

Low learning rates and frozen early layers reduce overwriting; progressive unfreezing combines both protections. If old-task performance must be retained, also mix old examples into training, use distillation or parameter regularisation, or keep separate task heads.

The BatchNorm footgun

Batch Normalization, or BatchNorm, uses mini-batch statistics during training and stored running mean and variance during evaluation. Those running estimates are buffers, not ordinary trainable weights.

Setting requires_grad = False prevents gradient updates to BatchNorm’s scale and shift, but does not necessarily stop running statistics changing in training mode. A frozen ImageNet backbone trained with small batches of dark microscope images can replace useful statistics with noisy estimates. The representation then changes underneath the head, producing a train/evaluation gap.

In correct evaluation mode, changing validation batch size should not materially change predictions. If it does, inspect whether model.eval() was called; BatchNorm may still be using batch statistics.

For a frozen backbone, use model.train(); backbone.eval() during head training. The head remains in training mode, while the backbone’s BatchNorm and other training-only behaviour stay fixed. During validation, call model.eval().

When fully fine-tuning with a large, representative dataset, updating BatchNorm statistics may help. With small data or tiny batches, frozen statistics are often safer. Make the choice deliberately.

A correct PyTorch starting point

This example uses pretrained ResNet-18 weights, replaces the ImageNet classifier with a two-class head, freezes the feature extractor, and trains one batch. Real images must use the weights’ matching resize and normalization.

import torch

from torch import nn
from torchvision.models import ResNet18_Weights, resnet18

weights = ResNet18_Weights.DEFAULT
base = resnet18(weights=weights)

# Everything except the original ImageNet classifier is the backbone.
backbone = nn.Sequential(*list(base.children())[:-1])
head = nn.Linear(base.fc.in_features, 2)
model = nn.Sequential(backbone, nn.Flatten(), head)

for parameter in backbone.parameters():
    parameter.requires_grad = False

# Keep frozen-backbone BatchNorm layers in eval mode.
model.train()
backbone.eval()

optimizer = torch.optim.AdamW(head.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

x = torch.randn(8, 3, 224, 224)
y = torch.randint(0, 2, (8,))

logits = model(x)
loss = loss_fn(logits, y)

optimizer.zero_grad()
loss.backward()
optimizer.step()

print(logits.shape)

The final line prints:

torch.Size([8, 2])

For real training, replace x and y with dataset batches. Keep integer class labels and raw logits with CrossEntropyLoss; do not apply softmax first.

To fine-tune, make selected backbone parameters trainable and use optimiser parameter groups with a smaller backbone learning rate. Inspect requires_grad before training.

Failure modes you will see first

Symptom	Likely cause	Fix
Validation is near chance	Input size, channels, or normalization do not match pretrained weights	Use the weights’ documented transforms consistently
Training reaches 100% while validation falls	Too much adaptation for too little data	Return to a linear probe, unfreeze later blocks only, and lower the backbone rate
Training and evaluation differ sharply	BatchNorm statistics or evaluation mode is wrong	Use backbone.eval() for a frozen backbone and model.eval() for validation
A linear probe cannot beat the majority baseline	Representation, preprocessing, or labels are faulty	Inspect examples and labels before trying more layers

A practical recipe

For the solder project:

1. Split by physical board, not by near-duplicate photographs, to prevent leakage.
2. Load pretrained weights with their matching preprocessing.
3. Train a new head and record validation metrics plus a majority-class baseline.
4. If useful but insufficient, progressively unfreeze later blocks with a backbone rate roughly ten times smaller than the head’s. Stop when validation stops improving.
5. Check trainable parameter counts, evaluation mode, and BatchNorm behaviour. Choose hyperparameters on validation data and use the test set once.

The honest limitation

Transfer learning cannot manufacture information. If a crack is smaller than the image resolution, more fine-tuning will not help. If labels disagree, the model learns that disagreement. A radically different sensor can make pretrained features a poor starting point.

A domain-specific model trained from scratch can win when you have enough varied data and compute. Domain-adaptive pretraining—training on many unlabelled images from the new domain before fine-tuning—can be a useful middle path.

The same early-versus-late idea appears in language models, but transformer weights, tokenisation, context length, and memory costs change the trade-offs. See fine-tuning for LLMs and LoRA.

What to remember

• Transfer learning reuses a pretrained backbone; fine-tuning updates some or all of it.
• Early layers tend to learn reusable local patterns; later layers are more task-specific.
• Start small and similar with a linear probe; adapt more for large or different datasets.
• Give the random head a larger learning rate than the pretrained backbone.
• Frozen weights do not automatically freeze BatchNorm statistics.

Quick check

0/3

Q1

The head contains no parameters, so the optimiser compensates with a larger rateThe head starts with random weights and needs large corrections, while the backbone is already near a useful representationThe backbone never receives gradients during fine-tuningA larger rate prevents BatchNorm statistics from changing

Check answer

Q2

Whether model.eval() was called before validation, so BatchNorm uses stored running statisticsWhether the final classifier has too many output classesWhether the optimiser is using momentumWhether the images contain exactly 224 pixels

Check answer

Q3

Freeze every layer permanently because the dataset is largeTrain only a linear head and never compare another strategyCompare full or progressive fine-tuning with a small backbone learning rate, and consider domain-adaptive pretrainingDiscard the pretrained model automatically because thermal images are different

Check answer