What is the vanishing gradient problem and how do you fix it?

How to think about it

The vanishing gradient problem occurs when repeated multiplication during backpropagation makes gradients shrink toward zero, so early layers receive almost no learning signal. I usually address it with non-saturating hidden activations, suitable initialization, normalization, and residual connections, while checking whether the real problem is exploding gradients or a broken training graph.

Why the gradient disappears

Backpropagation applies the chain rule to move the loss gradient from the output back toward the input. The loss is the number measuring how wrong the model’s prediction was. Each layer transforms the gradient before passing it to the previous layer.

A Jacobian is the matrix of partial derivatives describing how one layer’s outputs change when its inputs change. Schematically, the gradient reaching an early layer is

g_l = J_(l+1)^T J_(l+2)^T ... J_L^T g_L

where J_k is the Jacobian for layer k, and g_L is the gradient at the output. The gradient of that layer’s weights also depends on the activation entering the layer.

If most of those transformations shrink vectors, their product shrinks rapidly. A factor of 0.9 repeated 100 times is about 0.000027. A factor of 0.25 repeated 20 times is already less than one trillionth. That is why depth matters: the problem is multiplicative, not additive.

The sigmoid activation illustrates the issue. Its derivative is

sigmoid'(z) = sigmoid(z) * (1 - sigmoid(z))

and its largest possible value is 0.25, at z = 0. For large positive or negative values, sigmoid saturates, meaning its output is on a flat part of the curve and its derivative is close to zero. A saturated unit may output something sensible, but it passes almost no information about how to improve.

The tanh activation, or hyperbolic tangent, has a similar problem. Its derivative is 1 - tanh(z)^2. It is close to one near zero and close to zero when the activation approaches either -1 or 1.

The weight matrices matter too. Even with ReLU, a layer can shrink or enlarge a gradient depending on its weights. In linear-algebra terms, if the typical stretching factor of each layer is below one, the gradient tends to vanish; if it is above one, it tends to explode. The activation function is only one part of the chain.

This is also why vanilla recurrent neural networks are vulnerable. The same transformation is applied repeatedly across time steps, so a sequence of 100 tokens can create the same kind of long multiplication chain as a very deep feed-forward network.

A concrete calculation

Imagine a fraud detector with 120 normalized transaction features and 20 hidden layers, all using sigmoid activations. This is a deliberately poor design, but it makes the arithmetic visible.

Even using the maximum sigmoid derivative at every layer, the activation part of one backward path is:

derivative = 0.25
depth = 20
signal = derivative ** depth

print(f"{signal:.3e}")

The output is:

9.095e-13

That is an idealized upper-bound calculation. Real networks contain many paths, and their weight matrices may amplify or shrink the signal as well. If the pre-activation is around z = 3, the sigmoid derivative is only about 0.0452, so that particular path shrinks much more severely.

Suppose plain stochastic gradient descent sees a parameter gradient of 1e-12 and uses a learning rate of 1e-3. The raw parameter step is 1e-15. A weight whose magnitude is 1e-2 will barely move. Meanwhile, the last few layers may still have gradients around 1e-2 because their paths to the loss are short. The model appears to train, but only its upper layers are doing the work.

That pattern is usually the first practical clue: the loss falls a little, then plateaus, while the earliest layers remain almost unchanged.

How I would fix it

1. Use an activation with a healthier derivative

For hidden layers, I would normally replace sigmoid with ReLU, the rectified linear unit defined as max(0, x), or with GELU, the Gaussian Error Linear Unit.

ReLU has derivative one on its positive side, so an active unit does not automatically multiply the gradient by a number such as 0.25. GELU is smoother and is common in modern transformer-style networks.

Neither is magic. ReLU’s derivative is zero for negative inputs. If a unit receives negative values on nearly every example, it can become a “dead ReLU” and stop learning. Leaky ReLU gives the negative side a small nonzero slope, which can help in that case. GELU can also have small derivatives for strongly negative inputs.

I would usually change hidden activations, not blindly remove a sigmoid output. A sigmoid output can be appropriate for a binary probability. The important question is whether hidden layers are saturating and starving earlier layers.

2. Match initialization to the activation

Initialization controls the starting scale of weights. If weights are too small, every layer may shrink signals before training has a chance to correct them. If they are too large, activations can saturate or gradients can explode.

He, also called Kaiming, initialization is designed for ReLU-like activations. A common normal version uses a standard deviation near

平方根(2 / fan_in)

where fan_in is the number of inputs to a neuron. In ordinary notation, that is sqrt(2 / fan_in).

Xavier, or Glorot, initialization is commonly used for tanh and roughly linear layers. Its normal version uses a standard deviation near

sqrt(2 / (fan_in + fan_out))

These choices aim to keep activation and gradient variance from changing dramatically at initialization. They do not guarantee healthy gradients after training starts. A network can begin well and later drift into saturation, which is why initialization and monitoring belong together.

3. Add residual connections

A residual connection, also called a skip connection, adds a layer’s input directly to its transformed output:

y = x + F(x)

The derivative is

dy/dx = I + J_F

where I is the identity matrix, which leaves a vector unchanged. The identity term gives the gradient a direct route around F. It does not have to survive only through every multiplication inside the residual branch.

A simple residual multilayer block looks like this:

import torch.nn as nn
import torch.nn.functional as F

class ResidualMLP(nn.Module):
    def __init__(self, width):
        super().__init__()
        self.norm = nn.LayerNorm(width)
        self.up = nn.Linear(width, 4 * width)
        self.down = nn.Linear(4 * width, width)

    def forward(self, x):
        return x + self.down(F.gelu(self.up(self.norm(x))))

The input and output widths match, so the addition is valid. If they do not match, the skip path needs a learned projection.

The important nuance is that a residual connection does not make the entire gradient equal to one. The residual branch can still have poorly scaled weights, and repeated residual blocks can still become unstable. It provides a highway; it does not repair every road in the city.

4. Normalize activations when the architecture supports it

Batch normalization, or BatchNorm, normalizes each feature using statistics computed across a small batch of examples. This can keep pre-activation values in a range where nonlinearities are not constantly saturated.

Layer normalization, or LayerNorm, normalizes features within each individual example. It does not depend on statistics from other examples, which makes it a better fit for variable-length sequences and many transformer architectures.

The trade-off matters. BatchNorm often works well in convolutional networks with reasonably large, representative batches, but tiny batches make its statistics noisy. It also has different training and inference behavior because inference uses stored running statistics. LayerNorm avoids those batch issues, but it is not automatically better for every convolutional model and cannot compensate for a badly designed network.

In many deep sequence models, a pre-normalized residual block is a practical pattern: normalize first, apply the transformation, then add the skip connection.

5. Use an architecture with shorter or additive paths

For recurrent models, LSTMs and GRUs add gates that regulate information flow. An LSTM’s cell state provides a comparatively direct additive path across time, which is much easier to preserve than repeatedly passing information through a plain nonlinear recurrence.

For very deep feed-forward and attention models, residual connections and normalization serve a similar purpose. The goal is not merely to choose a fashionable activation. It is to make the route from the loss to early parameters numerically survivable.

Gradient clipping belongs in the same conversation, but it fixes the opposite problem. Clipping limits an excessively large gradient, often to prevent NaN values. It cannot recreate a gradient that has already shrunk to zero.

How I would diagnose it

I would inspect gradients by layer after loss.backward() and before the optimizer clears them:

loss.backward()

for name, parameter in model.named_parameters():
    if parameter.grad is not None:
        print(f"{name}: {parameter.grad.norm().item():.3e}")

A typical vanishing-gradient symptom is early-layer norms around 1e-9 while final-layer norms are around 1e-2, together with a loss that stops improving. I would also inspect activation distributions: sigmoid outputs clustered near 0 and 1, or tanh outputs clustered near -1 and 1, indicate saturation.

Compare gradient size with parameter size as well. A raw gradient norm depends on layer width, so the update-to-weight ratio is often more informative.

If every layer has zero gradients, I would not immediately blame depth. I would check whether the parameters are included in the optimizer, whether the loss is connected to the model output, and whether inference mode or no_grad was accidentally used. If gradients are infinite or NaN, I would investigate exploding gradients, learning rate, numerical overflow, and normalization instead.

What they’ll ask next

Does ReLU completely solve vanishing gradients?
No. It avoids sigmoid’s systematic shrinkage on positive inputs, but negative units have zero derivative and can die. Poor weight scaling and deep matrix products can still cause trouble. Residual connections, normalization, and suitable initialization address the remaining causes.

Do residual connections eliminate the problem?
No. They add an identity route, so the gradient can bypass the residual transformation, but the residual branch can still be unstable. Very deep residual networks still need sensible initialization, normalization, and monitoring.

Why not just increase the learning rate?
A larger learning rate multiplies whatever gradient exists; it does not restore information lost through a near-zero chain. It may make later layers unstable while early layers remain effectively frozen. Fix the gradient path first, then tune the learning rate.

Say this in the interview: The vanishing gradient problem is caused by repeated Jacobian products shrinking the backward signal, so early layers stop learning; I address it with suitable activations and initialization, normalization, residual or gated paths, and gradient monitoring rather than treating clipping as a universal fix.