What is gradient clipping and when would you use it?

How to think about it

Gradient clipping limits an update when the gradient becomes dangerously large, usually by rescaling the whole gradient vector to a maximum norm before the optimizer step. I use it as a guardrail against exploding or unusually spiky gradients, especially in long-sequence RNNs and unstable deep-model training; I do not treat it as a substitute for fixing the underlying instability.

Why gradients explode

A gradient is the vector of partial derivatives that tells the optimizer how the loss changes when each model parameter changes. If the gradient is g, the simplest update looks like θ ← θ − ηg, where θ is the parameter vector and η is the learning rate.

That update is only safe when g is reasonably sized. A single unusually large gradient can move the parameters far outside the region where the model has learned anything useful. The next forward pass may produce a much larger loss, followed by NaN or Inf values. In a training dashboard, the first visible symptoms are often a sudden loss spike and a gradient norm jumping from, say, 0.8 to 37, followed by failed optimizer steps.

The classic case is a recurrent neural network. During backpropagation through time, derivatives pass through the recurrent operation once for every time step. The resulting derivative contains repeated products of Jacobian matrices. If their effective scale is consistently above one, the product grows rapidly; if it is below one, it shrinks toward zero. The first case is exploding gradients. The second is vanishing gradients.

Transformers do not have the same recurrent loop, but they can still produce gradient spikes. Common causes include an excessive learning rate, unstable initialization, poorly scaled residual paths, problematic data, very large batches, or numerical trouble in mixed-precision training. Clipping is useful in those situations because it limits the damage from one bad step.

It changes the update, not the forward pass and not the loss itself. If the gradient is below the limit, clipping does nothing. If it is above the limit, clipping deliberately takes a smaller step.

A concrete example

Suppose a GRU is learning to predict the next token in customer-support messages as long as 2,048 tokens. At step 842, the logs show:

• loss: 2.3, then 18.0
• pre-clipping gradient norm: 37
• learning rate: 0.01

With plain stochastic gradient descent, the update has norm approximately 0.01 × 37 = 0.37. That is a very large move compared with the normal updates in this run. If the maximum global norm is 1.0, clipping scales the gradient by 1 / 37. The resulting update has norm approximately 0.01 × 1 = 0.01.

The step is not magically correct. It is simply prevented from being catastrophic. Training can continue, giving you time to fix the sequence length, learning rate, initialization, or other cause.

For a small gradient vector, suppose g = [3, 4]. Its Euclidean, or L2, norm is 5. With a maximum norm of 2, global-norm clipping multiplies every component by 2 / 5, producing [1.2, 1.6]. The direction is preserved; only the length changes.

That direction-preserving property is why global-norm clipping is usually the default choice.

Norm clipping versus value clipping

There are two common forms.

Method	Operation	Main trade-off
Global-norm clipping	Rescale the complete gradient when its norm exceeds a threshold	Preserves direction, but one large step can still affect every parameter
Value clipping	Replace each component with a value inside a fixed range	Simple, but can distort the gradient direction

With value clipping at 2, the vector [3, 4] becomes [2, 2]. Both components are cut independently, so the direction changes substantially. A vector that originally said “move more strongly in the second coordinate” now says “move equally in both.”

Value clipping can be appropriate when individual coordinates are the problem or when a particular framework and existing recipe use it. For most neural-network training, I would start with global-norm clipping. Per-layer norm clipping is another option, but it changes the relative scale between layers because each layer gets its own limit. That can be useful when one small subsystem repeatedly dominates the update, but it is a different intervention from global clipping.

Common misconception: A maximum gradient norm of 1.0 does not mean every parameter changes by at most 1.0. It limits the norm of the gradient before the optimizer transforms it. The learning rate, Adam-style adaptation, momentum, and weight decay still affect the parameter update.

The production pattern

In PyTorch, the usual order is backward pass, clip, optimizer step:

optimizer.zero_grad(set_to_none=True)

loss = loss_fn(model(x), y)
loss.backward()

pre_clip_norm = torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0,
)

optimizer.step()

clip_grad_norm_ modifies the gradients in place and returns the total norm calculated before clipping. Logging pre_clip_norm tells you whether clipping is actually active. The fraction of steps where that value exceeds 1.0 is also useful.

With automatic mixed precision, gradients are initially scaled to avoid underflow. They must be unscaled before clipping, otherwise the clipping threshold is applied to the artificially scaled values:

optimizer.zero_grad(set_to_none=True)

loss = loss_fn(model(x), y)
scaler.scale(loss).backward()
scaler.unscale_(optimizer)

pre_clip_norm = torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0,
)

scaler.step(optimizer)
scaler.update()

When accumulating gradients over several microbatches, clip after the final backward pass and immediately before the optimizer step. Clipping every microbatch is possible, but it changes the sum of the gradients: each contribution is constrained separately rather than the accumulated update being constrained once.

When I would use it, and when I would not

I would add clipping when:

• training a long-sequence RNN, LSTM, or GRU;
• observing occasional gradient-norm spikes;
• training becomes unstable even though most batches behave normally;
• a large model occasionally takes a destructive step during warm-up or after a learning-rate change;
• a recurrent or generative model must tolerate unusually difficult examples.

I would not add clipping and stop investigating if the gradient is nonfinite on every step. Clipping cannot turn NaN into a useful direction. I would inspect the loss computation, input data, initialization, learning rate, normalization, mixed-precision scaling, and any custom kernels.

The threshold is a hyperparameter, not a universal constant. 1.0 is a reasonable starting point and appears in many training recipes, but the right value depends on model size, batch size, whether the loss is averaged or summed, gradient accumulation, and optimizer. A threshold of 1.0 may be sensible for one model and so restrictive that it slows another.

The key diagnostic is how often clipping activates. If the pre-clipping norm is below the threshold on almost every step, clipping is mostly inactive, which is fine. If nearly every step is clipped, the threshold may be too low, or the model may have a deeper problem. Constant clipping can hide an excessive learning rate and can make learning unnecessarily slow because the optimizer repeatedly receives a truncated signal.

Clipping also interacts with adaptive optimizers. With Adam, clipping the raw gradient before optimizer.step() means Adam’s moment estimates are built from the clipped gradient. That can prevent one spike from contaminating the optimizer state. But it does not guarantee that the final Adam parameter update has norm 1.0, because Adam rescales coordinates and may apply weight decay separately. Gradient clipping and update clipping are not interchangeable.

What they will ask next

Should I clip by value or by norm?

Usually by global norm. It limits the size of the whole update while preserving its direction. Use value clipping when isolated coordinates are the specific failure mode or when reproducing a known recipe. Per-layer clipping is useful when layers have very different gradient scales, but it needs its own tuning.

Does gradient clipping solve exploding gradients?

It limits the consequence; it does not remove the cause. If clipping fires constantly, investigate the learning rate, initialization, sequence length, normalization, data outliers, and numerical precision. A model that only trains because every step is heavily clipped is still unhealthy.

Where does clipping go with Adam or mixed precision?

Clip the gradients after backward() and before optimizer.step(). With mixed precision, call the scaler’s unscale_ operation first. With accumulated gradients, clip once after accumulation and before the step.

Say this in the interview

“Gradient clipping caps the gradient norm before the optimizer update, usually by rescaling the entire vector, so one exploding gradient cannot destabilize training; I use it as a guardrail, monitor how often it activates, and still investigate the underlying learning-rate or numerical problem.”