How do LSTM gates solve the vanishing gradient problem?

How to think about it

When a sequence model must remember “not eligible” for 100 tokens, a vanilla RNN can reduce that earlier signal to almost nothing. An LSTM does not eliminate vanishing gradients, but it greatly reduces them by carrying a separate cell state through an additive, gate-controlled update; when the forget gate stays near one, the backward gradient has a near-identity path.

Why gradients vanish in a vanilla RNN

A gradient is the signal that tells training how much an earlier value contributed to the final error. If that signal becomes tiny, the optimizer has almost no useful information with which to change the parameters responsible for the earlier event.

A vanilla recurrent neural network repeatedly applies something like:

h_t = tanh(W_h h_(t-1) + W_x x_t + b)

Here, x_t is the input at time t, h_t is the hidden state, and tanh is a squashing function that compresses values into a limited range.

Suppose the model reads a sequence and the loss is calculated at step T. To update the hidden state at an earlier step k, backpropagation through time multiplies a chain of derivatives:

∂h_T/∂h_k = J_T J_(T-1) ... J_(k+1)

Each J is a Jacobian, meaning a matrix of local derivatives for one recurrent step. It contains the recurrent weight matrix and the derivative of tanh.

If the relevant scale of each step is about 0.9, then after 100 steps the signal is approximately:

0.9^100 ≈ 0.0000266

That is only about three hundredths of a thousandth of the original signal. The model may still technically contain information about the early token, but training cannot reliably learn how to use it.

The opposite problem is possible too. If the repeated scale is larger than one, the gradient can grow exponentially and cause unstable updates. Tanh makes matters worse when it saturates: for a large positive or negative input, its output is nearly flat, so its derivative is close to zero.

The core problem is repeated multiplication. The recurrence keeps transforming the same information through weights and nonlinearities, and a small loss at every step compounds into a very small gradient.

What the LSTM changes

A long short-term memory network, or LSTM, carries two states:

• The cell state C_t is the protected memory channel.
• The hidden state h_t is the exposed working state used to make predictions and calculate the next gates.

The cell state is updated by three learned gates. Each gate is a vector, so different memory coordinates can make different decisions at the same time.

Let u_t be the concatenation of the previous hidden state and current input:

u_t = [h_(t-1), x_t]

The gates and candidate memory are:

f_t       = sigmoid(W_f u_t + b_f)
i_t       = sigmoid(W_i u_t + b_i)
Ctilde_t  = tanh(W_C u_t + b_C)
o_t       = sigmoid(W_o u_t + b_o)

C_t       = f_t ⊙ C_(t-1) + i_t ⊙ Ctilde_t
h_t       = o_t ⊙ tanh(C_t)

The sigmoid function makes each gate value lie between zero and one. The symbol ⊙ means element-wise multiplication.

Component	What it controls
Forget gate f_t	How much old cell state to keep
Input gate i_t	How much new candidate information to write
Candidate Ctilde_t	The possible new content
Output gate o_t	How much cell state to expose as hidden state

The important operation is the cell update:

C_t = f_t ⊙ C_(t-1) + i_t ⊙ Ctilde_t

This is an additive update, not a complete replacement of the previous state. If the forget gate is near one and the input gate is near zero, the cell state is copied forward almost unchanged:

C_t ≈ C_(t-1)

For the direct memory path, the derivative is therefore:

∂C_t/∂C_(t-1) ≈ f_t

Across many steps, the direct path contributes a product of forget-gate values rather than a product of recurrent weight matrices and activation derivatives. If those forget gates stay near one, the product shrinks much more slowly.

That is the central answer an interviewer is looking for. The gates help because they create a controlled additive path through the cell state. The model can preserve information without forcing it through a new tanh transformation at every step.

A numerical example

Imagine a support-ticket classifier reading 102 tokens:

“The customer is not eligible … 100 more tokens of account history … refund.”

Suppose one cell-state coordinate has learned to represent eligibility. At token 2, the input gate writes:

C_2 = -1

Assume the negative value means “not eligible.” For the next 100 steps, the model decides that this memory is still relevant:

• Each forget-gate value is 0.99.
• Each input-gate value is close to zero.
• The candidate information is therefore mostly ignored.
• At the final step, the output gate is close to one.

The memory becomes:

C_102 ≈ -1 × 0.99^100 ≈ -0.366

The exposed hidden value is approximately:

h_102 ≈ tanh(-0.366) ≈ -0.350

The signal is weaker than the original, but it is still substantial.

Now compare a recurrent path with an effective scale of 0.9 over the same 100 transitions:

0.9^100 ≈ 0.0000266

That earlier “not eligible” signal is effectively gone from the gradient path. The numbers are illustrative rather than the output of a particular trained model, but they show why the additive carry matters.

The senior-level qualification

The phrase “LSTMs solve vanishing gradients” is too strong. They reduce the problem.

If the forget gate averages 0.95 for 100 steps, the direct gradient is still:

0.95^100 ≈ 0.00592

That is much better than 0.9^100, but it is still attenuation. If the forget gate is near zero, the cell deliberately erases the memory and the direct gradient through that coordinate is also near zero.

There are also additional gradient paths because the gates depend on h_(t-1), and the hidden state depends on the cell state. The simple product of forget gates describes the clean direct carry path, not every term in the full derivative. Those extra paths can still vanish or explode.

The sigmoid gates have their own limitation. The derivative of a sigmoid is at most 0.25 and becomes very small when the gate saturates near zero or one. This can make it slow to train the parameters that control a gate. It does not destroy the direct cell-state path, because that path uses the gate’s current value, f_t, rather than multiplying by the sigmoid derivative at every step.

Many LSTM implementations initialize the forget-gate bias positively. That gives the network an initial preference for retaining information, which is useful because forgetting everything at the start makes long dependencies hard to learn. It is only an initialization preference. Training can still drive the gate toward erasure.

Training procedure matters as well. With truncated backpropagation through time, a model may process 100 tokens but backpropagate through only the latest 50. If the hidden and cell states are detached between chunks, a loss at token 102 provides no gradient path back to the “not eligible” token at token 2. The LSTM can carry the value forward during inference, but the training signal cannot teach that dependency across the detached boundary.

There is a deployment trade-off too. An LSTM processes one time step after another, so its sequence computation is inherently sequential. That makes it less attractive for very long sequences when a transformer can connect distant tokens through self-attention and parallelize training more effectively. Standard full self-attention has its own cost as the context grows, however. LSTMs remain useful for streaming signals, small devices, bounded memory, and applications where maintaining one compact state per live sequence matters more than global access to the entire history. A self-attention model is not automatically the better engineering choice.

A failure mode you would actually observe

A common implementation bug is carrying C_t and h_t from one customer request into the next. The model then appears to have mysterious memory: predictions change when requests are reordered, and a ticket’s classification depends on the ticket processed immediately before it. Reset both states at sequence boundaries, or maintain separate state for each stream.

A modeling failure looks different. Accuracy may be strong when the decisive word appears within 10 tokens of the prediction, then collapse as the gap reaches 50 or 100 tokens. Inspect the forget-gate values and the training window. Low retention, aggressive truncation, or both may be cutting the useful path.

What they’ll ask next

Does an LSTM completely eliminate the vanishing gradient problem?

No. Its direct cell-state gradient is a product of forget-gate values. Values near one preserve the signal; values near zero erase it. LSTMs make the path easier to preserve, but they do not guarantee preservation over arbitrarily long sequences.

Why not set the forget gate permanently to one?

Because the model also needs to discard stale information. If a cell always keeps its old value, memories from one topic can contaminate later predictions. The forget gate allows the network to retain “not eligible” during the relevant part of a ticket and then reset that memory when a new sequence or topic begins.

How does a GRU compare with an LSTM?

A gated recurrent unit, or GRU, combines the cell and hidden state and uses update and reset gates rather than maintaining a separate cell state. It has fewer parameters and often performs comparably, but it gives the model a slightly less explicit separation between long-term storage and exposed output. The choice depends on validation quality, latency, memory, sequence length, and whether the application benefits from a separate persistent state.

Say this in the interview

“LSTMs reduce vanishing gradients by giving information a separate cell-state path with additive updates, so a forget gate near one creates a near-identity route for gradients; they reduce, but do not eliminate, vanishing gradients.”