What is gradient clipping, and when is it necessary?

How to think about it

Gradient clipping limits the size of gradients before the optimiser uses them. It is necessary when occasional gradient spikes produce unstable parameter updates; it is not a mandatory step for every neural network, and it should not be used to hide a broken training setup.

Why gradients become a problem

A gradient is the set of partial derivatives that tells the optimiser how changing each model parameter would change the loss. A large gradient therefore suggests a large change to the weights. With a learning rate of 0.001, a gradient is usually manageable. With a gradient 10,000 times larger on one batch, the resulting update can throw the model far away from the region where it was learning.

The classic example is a recurrent neural network, or RNN, which reuses a hidden state across a sequence. During backpropagation through time, the gradient is passed through the same recurrence again and again. By the chain rule, the result contains a product of many Jacobians, which are matrices of partial derivatives.

If those repeated transformations amplify the signal, the gradient grows exponentially with sequence length. If they shrink it, the gradient vanishes instead. The exact behaviour depends on the recurrent weights, activation derivatives, sequence length, and data. A simplified expression looks like a repeated product of terms such as WᵀD_t, where W is a recurrent weight matrix and D_t contains activation derivatives.

This is why a 1,000-step sequence can be much harder to train than ten separate 100-step sequences. The model has more opportunities to amplify a small numerical error.

Deep feedforward networks can also produce exploding gradients, particularly with poor initialisation, an aggressive learning rate, unusually large losses, or an outlier batch. Transformers do not have the same recurrent multiplication across time, so it is inaccurate to say that attention heads inherently require clipping. Deep transformer stacks, long contexts, large learning rates, unstable data, and numerical issues can still create gradient spikes.

The mechanism: scale the gradient, preserve its direction

The most common form is norm clipping. Imagine concatenating every parameter gradient into one long vector g. Its Euclidean norm is the vector’s overall size:

||g||₂ = sqrt(g₁² + g₂² + ... + gₙ²)

Choose a maximum norm C. If the norm is already below C, do nothing. If it is larger, multiply every gradient by the same factor:

g_clipped = g × C / ||g||₂

That matters because every component is scaled equally. The update becomes smaller, but its direction stays the same.

Here is a deliberately small example. Suppose the gradient is [3, 4]. Its norm is 5. With C = 1, the scale factor is 1 / 5, so the clipped gradient becomes [0.6, 0.8]. The direction is unchanged.

With ordinary stochastic gradient descent and a learning rate of 0.01, the unclipped update has norm 0.05; the clipped update has norm 0.01. The optimiser still moves in the direction suggested by the batch, but no single batch can demand a step larger than the chosen limit.

A concrete training example

Imagine a two-layer LSTM, a gated recurrent network, reading 1,024 telemetry readings from a factory machine and predicting its temperature ten minutes ahead.

Most batches produce a loss near 0.4 and a global gradient norm between 0.4 and 0.9. One batch contains a corrupted sensor sequence. Its loss jumps to 18.3, and the measured gradient norm is 12.5.

With max_norm=1.0, norm clipping scales every gradient by 1 / 12.5, or 0.08. If the optimiser were SGD with a learning rate of 0.001, the raw update would have norm 0.0125; the clipped update would have norm 0.001.

That single intervention may keep the model training instead of producing enormous weights followed by NaN losses.

In PyTorch, the usual pattern is:

import torch

for x, y in dataloader:
    optimizer.zero_grad(set_to_none=True)

    prediction, _ = model(x)
    loss = criterion(prediction, y)
    loss.backward()

    total_norm = torch.nn.utils.clip_grad_norm_(
        model.parameters(),
        max_norm=1.0,
    )

    optimizer.step()

clip_grad_norm_ modifies the gradients in place. Its return value is the total norm measured before clipping, which makes it useful for logging. If the returned value is 0.7, clipping did nothing. If it is 12.5, the gradients were scaled down.

With Adam, the arithmetic is less direct because Adam transforms each parameter’s gradient using running estimates of its first and second moments. Clipping still happens before that transformation. It limits the raw gradient that Adam receives; it does not guarantee that the final Adam parameter update has a particular norm.

Norm clipping versus value clipping

The other common method is value clipping, which clips each gradient element independently to a range such as [-1, 1].

Method	What it changes	Main advantage	Main drawback
Norm clipping	Scales all gradients together	Preserves direction	A single large coordinate can still dominate direction
Value clipping	Caps each element separately	Controls individual outliers	Distorts the gradient direction

For [3, 4] and a value limit of 1, value clipping produces [1, 1]. That is not the same direction as [3, 4]. Value clipping can be useful when individual coordinates are known to produce dangerous outliers, but global norm clipping is the usual first choice for neural-network training.

The order is part of the algorithm

The correct sequence is:

backward → clip → optimiser step

Clipping before backward() sees no newly computed gradients. Clipping after optimizer.step() is too late because the update has already happened.

Warning — order matters. The first symptom of incorrect clipping is often that the code appears to run while the gradient logs remain unchanged and the loss still occasionally becomes NaN. Put clipping after the final backward pass and immediately before the optimiser step.

Mixed-precision training adds one important detail. A gradient scaler may deliberately multiply the loss and gradients by a large scale factor to avoid underflow. Clipping those scaled gradients would compare them with the wrong threshold. Unscale first:

for x, y in dataloader:
    optimizer.zero_grad(set_to_none=True)

    with torch.autocast(device_type="cuda", dtype=torch.float16):
        prediction, _ = model(x)
        loss = criterion(prediction, y)

    scaler.scale(loss).backward()
    scaler.unscale_(optimizer)

    total_norm = torch.nn.utils.clip_grad_norm_(
        model.parameters(),
        max_norm=1.0,
    )

    scaler.step(optimizer)
    scaler.update()

Gradient accumulation also changes the placement. If eight microbatches are accumulated to make one optimiser update, clip after all eight backward passes, not after each one, unless per-microbatch clipping is an intentional design choice. Clipping each partial gradient changes the sum and can produce a different update from clipping the final accumulated gradient.

In distributed training, the relevant quantity is usually the global norm across all workers. Clipping each worker’s local gradient independently can produce a different result: one worker may see a small local norm while another sees a large one. Distributed data-parallel implementations commonly reduce gradients during backward, but sharded systems may require a framework-specific clipping utility that understands where the parameters and gradients live.

When clipping is useful, and when it is not

Clipping is especially useful when:

• training an RNN or LSTM over long sequences;
• the run shows occasional large gradient-norm spikes;
• a noisy or adversarial batch can create a very large loss;
• a large model is being trained close to numerical stability limits;
• gradient accumulation or distributed training makes rare spikes expensive;
• you need a cheap safety boundary around an otherwise healthy training job.

A maximum norm of 1.0 is a common starting point, not a universal constant. Log the unclipped norm during a pilot run. A threshold near the upper tail of healthy norms, perhaps around the 95th percentile, is a reasonable diagnostic starting point. Then tune it against training stability, convergence speed, and validation quality.

The threshold is not portable in a simple way. It depends on whether the loss is averaged or summed, the batch size, the number of accumulated microbatches, parameterisation, and the optimiser. A threshold of 1.0 in one model can represent a very different constraint in another.

Clipping is a poor substitute for fixing the cause. If the norm is above the threshold on nearly every step, the threshold may be too low, the learning rate may be too high, the inputs may be badly scaled, or the model may be poorly initialised. If the forward pass already produces Inf or NaN, clipping the backward gradient may be too late. Check the data, loss implementation, activation values, learning-rate schedule, and mixed-precision scaling.

There is also a cost. Clipping can suppress a genuinely useful large update, especially early in training. If the gradient norm is routinely clipped, learning may become slow and the model may underfit. A run that never clips is not automatically healthy, but a run that clips almost every step deserves investigation.

What they’ll ask next

Is gradient clipping just a lower learning rate?
No. Lowering the learning rate shrinks every update, including ordinary ones. Norm clipping leaves ordinary steps unchanged and scales only steps whose gradient norm exceeds the threshold. The two can be combined, but they solve different problems.

How do you know whether clipping is helping?
Log the pre-clipping norm, the fraction of steps clipped, the loss, and validation metrics. If clipping changes only rare spikes and removes NaN failures without hurting validation performance, it is doing useful guardrail work. If 80 percent of steps are clipped, investigate the learning rate, data scale, and threshold.

Why is norm clipping usually preferred to value clipping?
Norm clipping reduces the step while preserving the gradient direction. Value clipping treats every coordinate independently and can rotate that direction substantially. I would start with global norm clipping, then consider value or parameter-wise clipping only when the model or failure pattern justifies it.

Say this in the interview

“Gradient clipping limits the gradient magnitude before the optimiser step, usually by scaling the whole gradient vector when its norm exceeds a threshold; I use it as a guardrail for observed or likely gradient spikes, while still fixing the underlying learning-rate, data, or numerical-stability problem.”