Why do vanilla RNNs struggle with long sequences?

How to think about it

Give a vanilla RNN a 1,000-token review in which a word near the beginning changes the meaning near the end, and it has a difficult job. It struggles for two separate reasons: gradients are repeatedly multiplied as they travel backward through time, so they tend to vanish or explode, and the recurrent computation must process time steps in order, so the sequence cannot be parallelised across time.

Why the gradient disappears

A vanilla RNN keeps one evolving summary of the sequence, called its hidden state. At time step t, a common update is:

h_t = tanh(W_h h_{t-1} + W_x x_t + b)

Here, x_t is the current input, h_{t-1} is the previous hidden state, and W_h is the recurrent weight matrix. The tanh function squashes each value into the range from negative one to one.

That hidden state is also the model’s memory. If the RNN is reading a sentence, h_20 must carry whatever matters from the first 20 tokens into the next step. There is no separate memory cell and no direct connection from token 1 to token 1,000. Everything travels through the chain:

h_1 → h_2 → h_3 → ... → h_1000

During training, the model uses backpropagation through time, or BPTT, which means unrolling that chain and applying the chain rule backward through every step. Suppose the loss is L and we want to know how much an early state h_k affected the final loss. The gradient contains a product of one local derivative for every later time step:

∂L/∂h_k = ∂L/∂h_T × ∏ from t=k+1 to T of ∂h_t/∂h_{t-1}

Each local derivative contains the recurrent matrix and the derivative of tanh. With column vectors, it can be written as:

J_t = D_t W_h

where J_t is the local Jacobian, or matrix describing how a small change in the previous state changes the current state, and D_t contains the tanh derivatives.

This repeated multiplication is the heart of the problem. If the typical multiplier is smaller than one, the product shrinks exponentially. If it is larger than one in the relevant directions, the product grows exponentially.

A small calculation makes the scale clear:

for factor in (0.8, 1.2):
    print(f"{factor:.1f} -> {factor ** 99:.3e}")

It prints:

0.8 -> 2.546e-10
1.2 -> 6.901e+07

That is after only 99 repeated multiplications. A sequence of 1,000 steps gives the gradient many more opportunities to collapse or blow up.

The tanh term makes vanishing particularly common. Its derivative is 1 - tanh²(a), so it is at most one and becomes very small when the input is far from zero. At a = 3, the derivative is about 0.0099. A saturated unit therefore passes along only about one percent of a small change at that step. Several such steps can erase the learning signal before it reaches the early tokens.

The recurrent matrix matters too. Its spectral norm is the largest amount by which it can stretch a vector. If that norm is comfortably below one, the recurrent transformation is contractive: repeated application shrinks signals. If it is above one, some directions can grow. That does not mean every gradient will explode. The actual result depends on the direction of the gradient, the current hidden states, and the changing tanh derivatives. The useful interview answer is therefore not “a weight above one always explodes.” It is that long products of factors above or below one are numerically unstable.

A concrete 1,000-step example

Consider a simple copy task.

At step 1, the input contains the four-digit code 4827. Steps 2 through 999 contain filler words. At step 1,000, the input says: “Output the code.”

A 256-dimensional hidden state could represent 4827 in principle. The problem is getting the model to learn that representation and preserve it. The training signal for the output at step 1,000 must travel backward across 999 recurrent transitions before it can teach the model what to do at step 1.

If the gradient vanishes, the update associated with the first input is effectively zero. The model receives a useful signal about recent filler tokens, but almost no signal saying, “Remember the code from 999 steps ago.” It may learn shortcuts, such as emitting a frequent code, or simply behave as though the early input never existed.

This same shape appears in real text. A long customer-support ticket may introduce an account type near the beginning and state the relevant policy exception near the end. A classifier does not need to remember every word, but it may need to preserve one early fact for hundreds of steps. A vanilla RNN has to compress that fact into its hidden state and keep it from being overwritten.

There is no theorem saying a vanilla RNN can never learn the copy task. With carefully chosen weights, favorable data, short enough horizons, or specialized recurrent architectures, it can. The issue is that ordinary gradient-based training has to discover and maintain a long-lived signal through an unstable chain. “Can represent” and “can reliably train” are different claims.

The second problem: computation is serial

Even if the gradients were perfectly behaved, the forward pass has another limitation.

To calculate h_10, the model needs h_9. To calculate h_9, it needs h_8. A GPU can parallelise matrix operations inside one step, and it can process many sequences in a batch, but it cannot calculate all 1,000 hidden states independently because each state depends on the previous one.

So an RNN has a serial depth proportional to the sequence length. Training a long sequence also requires storing intermediate states for BPTT, because the backward pass needs them. Longer sequences therefore increase both wall-clock dependency and activation memory.

Self-attention changes this shape. During training, each position can read the other positions from the input sequence at the same layer, so positions can be processed in parallel. A token near the end can have a short computational path to a token near the beginning instead of crossing 999 recurrent transitions.

That advantage has a cost. Full self-attention creates pairwise interactions, so its time and memory cost grow roughly quadratically with sequence length. Also, an autoregressive Transformer still generates one new token at a time during decoding. Its training and prompt-processing paths are parallelisable; the generation loop is not magically simultaneous.

What helps, and what it does not fix

Technique	Main benefit	What remains
Gradient clipping	Limits damage from an exploding update	Does not restore vanished gradients or remove serial computation
Orthogonal initialization	Can make recurrent transformations preserve signal magnitude better	tanh saturation and long-range optimization are still problems
Truncated BPTT	Reduces training memory and compute	Dependencies outside the truncation window receive no direct gradient
LSTM or GRU	Adds learned routes for retaining or replacing information	The recurrent steps are still sequential
Self-attention	Shorter paths between distant positions and parallel training	Quadratic full-attention cost and sequential autoregressive generation

Gradient clipping is often described too generously. If the gradient norm is larger than a chosen threshold, the training system rescales it. A team might use a threshold of 5, but that number is a tuning choice, not a cure. Clipping can turn a disastrous update into a manageable one. It cannot make a gradient of 0.000000001 informative.

Truncated BPTT deliberately backpropagates through only the most recent K steps. It is useful when a stream is very long and the task mostly depends on recent history. But if K is 100 and the important dependency is 900 steps away, the model receives no gradient path connecting those two events. The hidden state may still be carried between chunks in the forward pass, but the learning signal is cut at the boundary.

LSTMs address the memory problem with a cell state and gates. A simplified cell update is:

c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t

The forget gate f_t decides how much old memory to retain, the input gate i_t decides how much new information to write, and c̃_t is a candidate update. The important structural change is the additive path from c_{t-1} to c_t. If the forget gate stays near one, information can pass through many steps without being multiplied by a fresh tanh derivative at every transition.

GRUs use a related idea with fewer gates. Neither architecture eliminates vanishing gradients entirely, and both still calculate one time step after another. They improve the memory path; they do not provide parallel recurrence.

The senior nuance: vanilla RNNs are not always wrong

The right conclusion is not “RNNs are useless.” It is “choose them when the dependency and serving pattern fit.”

A vanilla RNN can be attractive for a streaming sensor where events arrive one at a time and the system only needs a current anomaly score. Its inference state is one fixed-size vector, so memory does not grow with the number of events already seen. That matters on a small device and when retaining the entire history is impossible or unnecessary.

It is also reasonable as a small baseline when sequences are short and the task depends mostly on local context. If the maximum sequence length is 20 and the model is classifying short commands, the long-range gradient problem may never become the bottleneck.

The warning signs appear when performance depends on distance. A common first symptom is that validation looks good on short examples but degrades sharply when the important token is moved farther from the prediction. Training may plateau while recent-token tasks work normally. If gradients explode instead, the first visible symptom is often a sudden loss spike, NaN loss, or infinite parameter values. Logging gradient norms and evaluating accuracy by dependency distance can separate these cases.

The forward hidden state itself does not have to become enormous for an exploding gradient to occur. Because tanh bounds the hidden activations, the states may look perfectly ordinary while the derivatives used to update the weights become huge.

What they’ll ask next

Does gradient clipping solve the vanishing-gradient problem?

No. Clipping rescales gradients only when they are too large. It helps with exploding gradients and can stabilise optimization, but it cannot recover a learning signal that has already shrunk away. Vanishing gradients usually call for a different recurrent architecture, initialization strategy, sequence decomposition, or attention mechanism.

Why do LSTMs help if they are still RNNs?

They add a gated, mostly additive cell-state path. A forget gate near one allows information and gradients to pass through many steps with less distortion than repeatedly applying a nonlinear hidden-state update. LSTMs reduce the problem; they do not guarantee perfect long-term memory and they remain sequential.

Are Transformers always better?

No. They are usually better when long-range relationships and parallel training throughput matter, but full self-attention has quadratic cost in sequence length, and generation remains token by token. For a tiny streaming model with a fixed memory budget and short dependencies, a vanilla RNN or GRU may be the more practical choice.

Say this in the interview

“Vanilla RNNs struggle with long sequences because BPTT multiplies recurrent Jacobians, making gradients vanish or explode, while the hidden-state recurrence also forces sequential computation; LSTMs improve the memory path, and Transformers improve parallelism, but each brings its own trade-offs.”