Vanishing & exploding gradients

At 3 a.m., a training run looks healthy. The loss has fallen from 2.4 to 0.7. Then one step reports loss = NaN. The next checkpoint contains weights with values like 1.8e19, and the model is now a very expensive random-number generator.

The opposite failure is quieter. The loss moves for the first few layers of training, then plateaus. The final layer’s gradients are around 1e-2, but the first layer’s are 1e-12. Nothing crashes. The input-side layers simply stop learning.

Both failures come from the same place: backpropagation multiplies a chain of local derivatives. A gradient is the vector of partial derivatives saying how much each parameter would change the loss. If the factors in that chain are usually below one, the gradient vanishes; if they are above one, it explodes.

That is the problem. The useful engineering is finding which factor is misbehaving and fixing it, rather than clipping every symptom.

See it happen, layer by layer

The widget shows gradient norms across 15 layers. Gradients enter at the output and travel toward the input. A log-scale view matters: 1e-6, 1, and 1e6 cannot be shown honestly on an ordinary linear axis.

With sigmoid, the derivative never exceeds 0.25 and becomes much smaller when the unit saturates. Increasing the weight scale can make linear factors expansive, but can also push sigmoid units farther into saturation. The full chain depends on the complete local Jacobians, not on the activation derivative or weight scale alone.

Input-side layers are hit first by vanishing because their gradients have passed through every later layer.

The mechanism: a product of Jacobians

Take one layer:

z_l = W_l h_(l-1) + b_l

h_l = phi(z_l)

During backpropagation, the gradient entering the layer is multiplied by the weight transformation and the activation derivative:

d h_(l-1) = W_l^T (d h_l elementwise-multiplied-by phi'(z_l))

One layer applies one linear transformation and one local derivative. Ten layers apply ten such transformations, so the input gradient is a product of all the layer-specific transformations.

For a one-dimensional toy network, if each layer multiplies the gradient by a, then after L layers:

g_0 = g_L a^L

With a = 0.9, fifty layers leave 0.9^50, or about 0.00515, of the signal. With a = 1.1, fifty layers amplify it to about 117.4. These are gradient multipliers, not parameter updates: the optimizer later uses -\eta g, with the learning rate, optimizer state, parameter scale, and actual gradient also affecting the update.

Real networks use matrix-valued Jacobians. Their singular values describe direction-wise stretching, but the singular values of the full product matter: directions can rotate, align, or cancel between layers. Per-layer values alone cannot determine the result.

Sequence length creates the same problem. An RNN applies its transition once per time step, so 500 tokens can create a chain as long as a 500-layer network. LSTMs add gated paths, while attention provides shorter routes between distant positions. See RNNs and LSTMs.

Why activations and initialization matter

The local gradient scale is roughly the weight-Jacobian scale multiplied by the activation-derivative scale.

Sigmoid has derivative sigmoid(x) times (1 - sigmoid(x)), capped at 0.25. Far from zero it saturates, and ten layers with derivative around 0.1 contribute 0.1^10 = 1e-10.

Tanh has derivative one at zero but also saturates toward negative one and one. Large pre-activations therefore produce small gradients.

ReLU has derivative zero for negative inputs and one for positive inputs, so positive units do not shrink the gradient at the activation. But a unit whose pre-activation stays negative becomes dead: its output and gradient remain zero. Leaky ReLU avoids a completely zero negative slope; GELU and SiLU are smooth alternatives but can still have small derivatives.

Initialization keeps both forward activations and backward gradients near a useful scale. He initialization uses variance close to 2 / fan_in for ReLU-like activations. Xavier uses fan_in and fan_out and is often a better start for tanh or linear layers. These are statistical approximations, not guarantees after training begins. See weight initialization.

Why residual connections help

A plain stack computes:

h_(l+1) = f_l(h_l)

A residual block computes:

h_(l+1) = h_l + f_l(h_l)

Its derivative is:

I + J_f

The identity term creates a direct route for information and gradients, so the learned branch does not have to transport the entire signal. Residual networks can still become unstable if the residual branches grow too large; residuals make the healthy regime easier to reach, not immune to calculus.

Transformers combine residual paths with normalization and careful initialization. Pre-normalization puts normalization before a sublayer and often gives deep stacks a more stable route. LayerNorm and RMSNorm reduce activation-scale drift without strictly bounding every gradient. See Normalization layers for their differences.

Exploding gradients: clip gradients, not the diagnosis

Exploding gradients commonly involve recurrent models, oversized learning rates, poorly scaled weights, or unstable losses. Gradient clipping caps the raw gradient before the optimizer uses it; it does not generally cap the final parameter update because momentum, weight decay, and coordinate-wise preconditioning can change that update.

For global L2-norm clipping, treat all parameter gradients as one vector. If its norm is G and the maximum is M:

scale = min(1, M / G)

Multiply every gradient by that same scale. If G = 13.93 and M = 5, the scale is about 0.359, producing a norm of 5 while preserving the direction of the concatenated raw gradient.

import numpy as np

# Gradients after backward().
grads = [np.array([3.0, 4.0]), np.array([12.0, 0.0]), np.array([0.0, 5.0])]

global_norm = np.sqrt(sum((g**2).sum() for g in grads))
print(f"global grad norm = {global_norm:.2f}")

max_norm = 5.0
if global_norm > max_norm:
    scale = max_norm / (global_norm + 1e-6)
    grads = [g * scale for g in grads]
    print(f"clipped: scaled all grads by {scale:.3f}")

new_norm = np.sqrt(sum((g**2).sum() for g in grads))
print(f"new global norm = {new_norm:.2f}  (raw-gradient direction unchanged)")

global grad norm = 13.93
clipped: scaled all grads by 0.359
new global norm = 5.00  (raw-gradient direction unchanged)

In PyTorch, check the loss before backpropagation, then clip after backward() and before the optimizer step:

if not torch.isfinite(loss).all():
    optimizer.zero_grad(set_to_none=True)
    raise FloatingPointError("non-finite loss")

loss.backward()
total_norm = torch.nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0, error_if_nonfinite=True
)
optimizer.step()

A non-finite loss or gradient must stop the update. Clipping cannot turn NaN or infinity into a useful value; inspect the first non-finite tensor in the forward and backward computations.

With automatic mixed precision, gradients remain scaled until unscaled. If accumulating eight microbatches, unscale and clip once after all eight backward passes:

scaler.scale(loss).backward()  # repeat for each microbatch

# Once, after all accumulation microbatches:
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0, error_if_nonfinite=True
)
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad(set_to_none=True)

Log norms after unscale_, or you measure the artificial scale. Float16 commonly needs a scaler; bfloat16 generally does not because it has float32’s exponent width. Clipping each microbatch defines a different optimization rule by shrinking large microbatches before they combine.

Value clipping limits each component and can change the raw-gradient direction. Norm clipping usually preserves the direction of the concatenated raw gradient, so it is the safer default for occasional oversized gradients. Neither fixes a bad loss, invalid input, or a learning rate that is too large on every step.

Diagnose the first symptom

On the same step, record the loss, global gradient norm, and several per-layer norms. The global number is a smoke alarm, not a full diagnosis: one small layer can fail while the total remains ordinary.

• loss = NaN and norm inf: suspect overflow, an invalid input or logarithm, or an exploding backward pass. Check finiteness after forward and backward.
• A flat loss with late-layer norms near 1e-2 and first-layer norms near 1e-12: inspect saturation, initialization, and plain deep paths.
• ReLU channels with exactly zero output and gradient: inspect dead units, learning rate, and initialization.
• A single spike followed by recovery suggests an unusual batch or sequence. Clipping on nearly every step means the threshold is permanently changing the optimizer’s input; investigate scale, learning rate, and initialization.

After all backward passes—and after scaler.unscale_(optimizer) when applicable— inspect per-layer and global norms:

squared_norms = []
for name, parameter in model.named_parameters():
    if parameter.grad is not None:
        norm = parameter.grad.detach().norm()
        squared_norms.append(norm.square())
        print(name, norm.item())

global_norm = torch.stack(squared_norms).sum().sqrt() if squared_norms else 0
print("global grad norm =", global_norm.item() if hasattr(global_norm, "item") else global_norm)

Record several steps, and compare gradient norms with parameter norms. A gradient of 0.01 may be huge for a parameter of 0.001 and negligible for one of 100. Exactly zero gradients also warrant checks for detached tensors, frozen parameters, unused branches, or an activation that is zero on the current data.

Do not blame gradients for every unstable run. Adam or AdamW can be unstable with an ordinary raw norm if its learning rate is too high or the loss scale changes sharply. Conversely, a tidy clipped gradient does not prove that forward activations are finite. Inspect the training stages in order.

When not to use each fix

Use clipping for occasional oversized gradients with a sensible direction, especially in RNNs, variable-length sequences, or runs with rare hard batches. Do not make it the first response to a norm that is huge from the first step: that usually indicates initialization, architecture, normalization, loss scaling, or learning rate. Frequent clipping can hide the problem and slow learning.

Use activation or initialization changes when saturation or dead units appear, residual connections when a plain stack must transport signal across too much depth, and normalization when activation scale drifts. Stable gradient norms are necessary plumbing, not proof that the data, labels, loss, or target are correct.

Quick check

Next

Once gradient magnitudes are healthy, see how the optimizer turns them into useful steps: optimizers and learning-rate schedules.