What causes exploding gradients and how is gradient clipping a fix?

How to think about it

Exploding gradients happen when repeated chain-rule multiplications make the backward signal grow instead of shrink. Gradient clipping is a safety mechanism: it limits the gradient’s size before the optimizer updates the weights, usually without changing its direction. It prevents a catastrophic update, but it does not repair the underlying model or data problem.

Why gradients explode

A gradient tells us how much the loss would change if a parameter changed slightly. During backpropagation, that signal is passed through every layer between the loss and the parameter.

The derivative of one layer is represented by a Jacobian, a matrix describing how small changes in the layer’s inputs affect its outputs. Backpropagation multiplies these Jacobians together. In a recurrent neural network, or RNN, the same process happens across time steps. The model reuses a hidden state, so a signal may pass through the same recurrent transformation dozens or hundreds of times.

For an RNN with hidden state h_t = φ(W h_(t-1) + U x_t), the gradient from time T back to time k contains a product of Jacobians:

∂L/∂h_k = J_(k+1)^T J_(k+2)^T ... J_T^T ∂L/∂h_T

If the matrices in that product tend to stretch vectors, the gradient grows exponentially. If they tend to shrink vectors, the gradient vanishes.

The relevant measure is the spectral norm, the largest stretch factor of a matrix. It is the largest singular value, where a singular value tells us how much the matrix can stretch a particular direction. A spectral norm above one means expansion is possible. It does not mean every gradient will explode, because the actual gradient may point in a direction that is not strongly amplified. But repeated expansion is dangerous.

A simple one-dimensional example makes the mechanism obvious. Pretend one recurrent coordinate behaves like this:

h_t = w h_(t-1)

The gradient passing backward through T steps is multiplied by w^T.

Suppose a refund-ticket classifier has a recurrent weight of w = 1.2, and one useful signal must travel backward through 50 recurrent transitions. Then:

1.2^50 ≈ 9,100

If the gradient at the final step is only 0.1, the earlier gradient becomes roughly 910. With a learning rate of 0.001, ordinary stochastic gradient descent would try to move that parameter by about 0.91. If the parameter was around 0.2, that is not a small correction. It is a complete rewrite.

Nothing dramatic happened at one step. Each step amplified the signal by only 20 percent. Compounding is the culprit.

The same issue can occur in a deep feed-forward network. A long product of weight matrices and activation derivatives can amplify a signal across layers. RNNs are especially vulnerable because the product length grows with sequence length and the same weights are reused repeatedly.

What gradient clipping changes

Let g be the complete gradient vector, formed by treating all parameter gradients as one long vector. Let τ be the chosen maximum norm.

With global L2 norm clipping:

• If ||g||₂ is at most τ, leave the gradient alone.
• If ||g||₂ is greater than τ, replace it with g × τ / ||g||₂.

That means the entire vector is rescaled by one common factor. Its direction stays the same; only its length changes.

For example, suppose the gradient is [3, 4]. Its L2 norm is 5. With max_norm=1, clipping produces [0.6, 0.8]. The vector points in exactly the same direction, but its length is now 1.

With a learning rate of 0.1, plain SGD would make an update of [-0.06, -0.08] instead of [-0.3, -0.4]. The update is still following the model’s proposed direction. It is simply no longer allowed to take a five-times-larger jump because one backward pass produced an unusually large signal.

This is why global norm clipping is usually preferred to value clipping, which clips each gradient entry independently. Value clipping could turn [10, 0.1] into [1, 0.1]. That changes the vector’s direction, not merely its magnitude. Value clipping can be useful when individual coordinates are the problem, but it is a more aggressive distortion.

The word “global” matters. Clipping each parameter tensor separately allows the combined gradient across tensors to remain large. Global clipping controls the norm of the full model gradient.

A basic PyTorch pattern

Clipping belongs after backward() has produced gradients and before optimizer.step() consumes them.

for inputs, targets in loader:
    optimizer.zero_grad(set_to_none=True)

    outputs = model(inputs)
    loss = loss_fn(outputs, targets)
    loss.backward()

    grad_norm = torch.nn.utils.clip_grad_norm_(
        model.parameters(),
        max_norm=1.0,
    )

    if not torch.isfinite(grad_norm).item():
        raise RuntimeError("Non-finite gradient norm")

    optimizer.step()

clip_grad_norm_ modifies gradients in place. Its return value is the total norm before clipping, which makes it useful for monitoring. If the original norm was 37.4 and the threshold was 1.0, the function returns approximately 37.4, while the gradients used by optimizer.step() have norm approximately 1.0.

With automatic mixed precision and a gradient scaler, unscale the gradients before clipping:

scaler.scale(loss).backward()
scaler.unscale_(optimizer)

grad_norm = torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0,
)

scaler.step(optimizer)
scaler.update()

Clipping scaled gradients would apply the threshold to the artificial scale factor as well. The gradients could then be over-clipped after unscaling.

If using gradient accumulation, normally accumulate all microbatch gradients first and clip once before the optimizer step. For eight microbatches using mean losses, divide each loss by 8, call backward() eight times, then clip the accumulated gradient. Clipping each microbatch separately is not equivalent because clipping is nonlinear: the clipped sum can point in a different direction from the unclipped sum.

How I would diagnose it

A typical failure looks like this:

step 1838  loss=0.62  grad_norm=2.1
step 1839  loss=0.65  grad_norm=3.7
step 1840  loss=nan   grad_norm=248000.0

The first useful signal is usually the gradient norm immediately before the update. Also check whether the loss and activations were finite before backward, and whether parameters became non-finite after the optimizer step.

Do not treat a norm of 10 or 100 as a universal definition of exploding gradients. Norm scale depends on the number of parameters, batch size, whether the loss is summed or averaged, model architecture, and whether gradients are being accumulated. The useful question is whether the norm has an unusual spike relative to this model’s normal range and whether the resulting update is large relative to the weights.

A NaN loss does not prove that gradients exploded. Other causes include taking the logarithm of zero, dividing by zero, invalid labels, overflowing activations, corrupt input values, and numerical overflow in reduced-precision training. The diagnostic order should be:

1. Check that inputs, outputs, and loss are finite.
2. Check the gradient norm and individual gradients after backward.
3. Check parameter values after the optimizer step.
4. Compare the update size and clipping frequency with a healthy run.

Clipping is not the root-cause fix

Clipping is valuable, but it can hide a problem.

A lower learning rate reduces every update. Clipping only changes updates whose gradients exceed the threshold. If weights are growing because the learning rate is too high, clipping may keep training alive while leaving the model poorly conditioned.

Better initialization can reduce the chance of early instability. He initialization is designed for networks using ReLU-like activations, Xavier or Glorot initialization targets a sensible variance for several other activations, and orthogonal initialization can keep a recurrent linear transformation’s singular values near one. None of these guarantees stability after training begins.

For RNNs, LSTMs and GRUs add gates that learn how much information and gradient should pass through time. Truncated backpropagation through time limits how many steps the gradient crosses, trading long-range credit assignment for better stability and lower memory use.

Input scaling and outlier handling matter too. A single unusually large feature can create a large activation, which then creates a large downstream derivative. Normalizing features and inspecting extreme examples is often more useful than immediately lowering the clipping threshold.

A common failure mode after adding clipping is a loss that stops improving. The logs show the post-clipping norm at exactly 1.0 on nearly every step, while the pre-clipping norm is often 50 or 100. That usually means the threshold is too low or the underlying model is unstable. Clipping every step turns a warning system into the entire training strategy.

What they’ll ask next

Is gradient clipping the same as lowering the learning rate?

No. Lowering the learning rate scales every update, including healthy ones. Clipping leaves ordinary gradients unchanged and limits only unusually large gradients. The two can be combined, but they solve different problems.

Does clipping fix vanishing gradients too?

No. Clipping only caps gradients that are too large. It cannot make a gradient of 0.000001 useful. Vanishing gradients need different remedies, such as shorter dependency paths, residual connections, suitable initialization, gated recurrent units, normalization, or a different architecture.

How do you choose max_norm?

There is no universal correct value. I would log unclipped norms during a stable baseline, examine their distribution, and choose a threshold that catches rare spikes without clipping most steps. I would also monitor the fraction of steps that were clipped. If clipping happens on 80 percent of steps, I would investigate the learning rate, initialization, data scale, loss implementation, and architecture rather than simply raising the threshold.

Say this in the interview: Exploding gradients come from a product of Jacobians that amplifies the backward signal, and global norm clipping stabilizes training by rescaling oversized gradients before the update while preserving their direction; it is a guardrail, not a substitute for fixing the cause.