Debugging a training run

At 2:07 a.m., your image classifier has been running for six hours. The training loss is NaN. The GPU graph says 20 percent. Validation accuracy has been 71.4 percent for the last 40 epochs.

Those are three different failures. Or they may be one failure wearing three hats.

A training run is a chain:

data → model → loss → gradients → optimizer → updated parameters

A loss measures how wrong the model is. A gradient points toward increasing loss; an optimizer usually uses its negative, with additional scaling, to choose a parameter update. Debugging means finding the first broken link, not changing five hyperparameters and hoping the graph improves.

First move: overfit one batch

Take one batch of 32 examples and reuse those exact examples. Remove random augmentation. Temporarily disable dropout and weight decay. Train on that batch repeatedly.

With verified, compatible labels and a model and learning rate capable of fitting it, a sufficiently expressive model should drive its loss very low and its accuracy close to 100 percent:

Step	Loss	Accuracy
0	0.70	50%
50	0.18	94%
200	0.01	100%

The exact values vary; the direction should not.

This test removes most moving parts. The data, labels, and inputs stop changing, so the optimizer gets repeated chances to solve one small problem. If it cannot, inspect the batch, labels, model, loss, gradients, parameter updates, numerical precision, and optimization settings. That is evidence, not proof: contradictory labels, unavoidable noise, or an unsuitable architecture can also prevent fitting. If it succeeds, those parts work together on at least one batch; investigate the data pipeline, split, changing input distributions, schedule, memory pressure, or bugs triggered by particular examples.

Log the gradient norm before clipping. The gradient norm summarises the size of all parameter gradients: zero means no update signal; a huge or non-finite value suggests instability.

import torch

torch.manual_seed(7)

x = torch.randn(32, 2)
y = (x[:, 0] + 0.5 * x[:, 1] > 0).long()

model = torch.nn.Sequential(
    torch.nn.Linear(2, 32),
    torch.nn.ReLU(),
    torch.nn.Linear(32, 2),
)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
loss_fn = torch.nn.CrossEntropyLoss()

for step in range(201):
    optimizer.zero_grad(set_to_none=True)
    logits = model(x)
    loss = loss_fn(logits, y)

    if not loss.requires_grad:
        raise RuntimeError(
            "Loss is detached: loss.requires_grad is False; "
            "inspect the forward graph and frozen parameters."
        )

    loss.backward()

    grads = [
        p.grad.detach()
        for p in model.parameters()
        if p.grad is not None
    ]
    if not grads:
        raise RuntimeError(
            "Backward produced no parameter gradients; "
            "inspect requires_grad, detach, and the optimizer's parameters."
        )

    squared_norm = torch.zeros((), device=loss.device, dtype=torch.float32)
    for grad in grads:
        squared_norm += grad.float().pow(2).sum()
    grad_norm = squared_norm.sqrt()

    if step % 50 == 0:
        accuracy = (logits.argmax(dim=1) == y).float().mean()
        print(
            f"step={step:3d} loss={loss.item():.4f} "
            f"accuracy={accuracy.item():.2f} grad_norm={grad_norm.item():.4f}"
        )

    optimizer.step()

If the test fails, compare the loss, gradient norm, and parameter change at one step. A finite loss with zero gradients suggests a detached graph or requires_grad problem. Nonzero gradients with no parameter change suggests a disconnected optimizer, zero learning rate, or frozen parameters. A non-finite gradient explains a non-finite update.

Do not leave every diagnostic change enabled. This test is a microscope, not a training recipe.

Symptom: loss is NaN or Inf

NaN means “not a number”; Inf means an infinite floating-point value. Once either enters the loss, it usually spreads through gradients and parameters. Continuing to train writes corrupted weights to disk.

Check these causes in order:

1. The batch is already non-finite. A corrupt image or division by zero during feature construction can do this. Check inputs and targets immediately after loading. Invalid targets are a separate contract problem: CrossEntropyLoss commonly raises an exception or device-side assertion, while other invalid targets may produce a finite but meaningless loss.
2. The forward pass or loss is numerically unstable. Logarithms of zero, large exponentials, and division by tiny values are common causes.
3. Gradients exploded. A large learning rate, poor initialisation, long sequences, or an unstable architecture can overflow. See vanishing and exploding gradients.
4. Half precision overflowed. FP16 has a smaller numeric range than FP32. Compare with a short FP32 run using mixed precision training.

Assert finiteness after loading the batch, after the forward pass, after the loss, and after backward. The first failure is the boundary to investigate. If step 37 is the first failure, log the gradient norm before clipping and the learning rate: a norm growing from 4 to 800 to Inf suggests exploding gradients; a normal norm followed by a bad loss points to loss arithmetic or a particular input.

Fix the boundary you found: reject bad samples, use stable library losses, lower the learning rate, correct initialisation or normalisation, or run the suspect operation in FP32.

Symptom: loss does not move

A loss stuck at 0.693 in a balanced two-class problem is roughly random guessing. A value stuck at exactly the same many-decimal value usually means the model is not changing.

Check:

• The update path: the optimizer may contain a different model, parameters may be frozen, or the learning rate may be zero.
• The computation graph: .detach(), torch.no_grad(), or a NumPy conversion can remove the path needed for backpropagation.
• Labels and outputs: constant or shuffled labels, wrong class indices, and an incorrect output/loss pairing can prevent learning. Do not apply an extra softmax before CrossEntropyLoss.
• The actual schedule: too small a learning rate looks frozen, and a scheduler may have reduced it earlier than intended. Read the learning-rate schedule used by the run.

Run the fixed-batch test and print the gradient norm, learning rate, and absolute difference in one parameter tensor before and after optimizer.step(). Zero gradients point to the graph or frozen parameters; nonzero gradients with no parameter change point to the optimizer or learning rate; changing parameters with a flat loss points to labels, dimensions, or the loss contract.

Symptom: training loss falls but validation does not

This is often overfitting: the model memorises training-specific details that do not transfer to unseen examples. It can also result from different preprocessing, incorrect evaluation mode, or a faulty split.

Check that training and validation use compatible normalisation, resizing, tokenisation, and label mappings. Use model.eval() for validation so dropout is disabled and BatchNorm uses evaluation behaviour. Verify that predictions align with labels and that the validation set is the intended data. The bias–variance view of overfitting gives more context.

Evaluate a few training examples in evaluation mode with the same metric used for validation. Good training performance but poor validation performance supports overfitting or distribution mismatch. Poor performance on both suggests evaluation behaviour or preprocessing is wrong. Possible fixes for genuine overfitting include more data, regularisation, earlier stopping, or a smaller model.

Symptom: validation is better than training

The two measurements may use different conditions. Training can include dropout, augmentation, noisy inputs, and BatchNorm’s mini-batch statistics. Validation usually uses clean inputs and evaluation mode. Training loss may also be averaged during updates, while validation is measured after the epoch.

Other possibilities are a genuinely harder training set, leakage or selection of the validation set, or metrics that are not comparable.

At the end of an epoch, evaluate a fixed training sample with model.eval(), no augmentation, and the exact validation metric. Compare it with validation under the same conditions. Do not “fix” a healthy gap merely because validation is higher.

Symptom: GPU utilisation is low

The GPU waits while the CPU decodes images, runs Python transforms, or copies a batch across the host-to-device boundary. A data-loader worker prepares batches in a separate process. A host-to-device copy moves data from CPU memory to GPU memory.

Time batch fetching separately from GPU computation. CUDA operations are asynchronous, so call torch.cuda.synchronize() immediately before and after the timed GPU region. If fetching takes 80 milliseconds and computation takes 10, model tuning will not help.

Common causes are slow transforms or storage, too few workers, synchronous copies, a workload that is too small, and unnecessary synchronisation from frequent .item(), printing, or explicit CUDA waits. Compare the real loader with pre-generated tensors already on the GPU. A large speedup identifies the input path as the bottleneck. Then test worker count, batch size, transforms, storage, and pinned memory. pin_memory=True and suitable non_blocking=True copies can help transfers, but cannot make a slow disk faster.

nvidia-smi is a sampled alarm, not a profiler.

Symptom: out-of-memory at a random step

A “random” OOM often follows a longer sequence, larger image, or batch with more detected objects. Memory can also accumulate gradually when computation graphs are retained.

Log the failing batch’s dimensions or token count, plus torch.cuda.memory_allocated() and torch.cuda.memory_reserved() before and after the step. A jump only on large examples suggests variable-size input; steady growth suggests retained graphs.

Frequent causes and fixes:

• Appending loss tensors to a list retains their graphs; use loss.item() or loss.detach().
• Validation or generation should use torch.no_grad() or torch.inference_mode().
• Reduce the microbatch size, then use gradient accumulation to recover the effective batch size. Mixed precision and activation checkpointing can reduce peak memory; see activation checkpointing.
• Cap or bucket variable lengths. Also check allocator fragmentation and other processes using the GPU.

Symptom: results are not reproducible

A seed repeats a random-number sequence; it does not turn a GPU into exact arithmetic.

Seed Python’s random, NumPy, PyTorch, and data-loader workers. Some GPU operations remain nondeterministic because parallel reductions and atomic updates can occur in different orders. Since floating-point addition is not associative, small differences can eventually lead to different optimisation paths.

A reproducible checkpoint also needs optimizer and scheduler state, mixed-precision scaler state when used, and often RNG states. Hardware, CUDA libraries, PyTorch versions, compiler choices, and distributed reduction order can matter.

Run twice with the same seed and compare the first batch’s IDs, first logits, loss, gradient norm, and parameters after one optimizer step. The first difference identifies the boundary. For controlled debugging, request deterministic algorithms where supported:

torch.use_deterministic_algorithms(True)

This can raise errors for unsupported operations, slow training, and still cannot promise identical results across platforms or software releases. Determinism is a debugging setting with a bill attached.

A small diagnostic ledger

Record every step or every few steps:

• loss, learning rate, and gradient norm before clipping;
• batch IDs, shape or token count, and data-wait time;
• parameter or update norms;
• GPU allocated and reserved memory;
• examples per second, evaluation mode, checkpoint, and seed.

Save the first bad batch and last healthy checkpoint. Change one variable at a time, starting with the fixed-batch test. Keep forward, loss, backward, clipping, update, and logging visibly separate in the training loop.

Decision table: which check earns the next ten minutes?

Check	Separates	Cost	Use it when
Overfit one fixed batch	Core model/update path vs changing data or configuration	Low	Loss is flat, strange, or unstable
Compare gradients and parameter delta	No signal vs blocked optimiser update	Very low	Loss is exactly unchanged
Re-run briefly in FP32	Numerical precision vs model or data	Low	NaN appears only with mixed precision
Compare fetch time with compute time	Input pipeline vs GPU workload	Low	GPU utilisation is low
Log shape and memory per batch	Variable input vs retained graph	Low	OOM appears at a particular step
Compare first batch and first update twice	RNG/data order vs nondeterministic arithmetic	Medium	Results differ between runs

No test proves the entire system correct. Each isolates a different boundary so you can stop guessing.

What to remember

• Overfit one fixed batch first.
• Find the first non-finite tensor.
• Use gradient norm, learning rate, and parameter delta to explain a flat loss.
• Time data fetching and GPU work separately.
• Reproducibility requires seeds, algorithm choices, complete checkpoint state, and a matching environment.

Quick check

0/3

Q1

The validation splitWhether the graph is detached or parameters are frozenThe number of data-loader workersWhether to add more dropout

Check answer

Q2

Validation always has more examplesThe optimiser secretly trains on validation examplesTraining may use dropout and augmentation, while validation uses clean inputs and evaluation modeAccuracy is mathematically larger on validation data

Check answer

Q3

Check the random seed; use a larger learning rateLog batch token counts and memory; bucket or cap sequence lengthsDisable deterministic algorithms; add dropoutEvaluate on the validation set; increase weight decay

Check answer