RNNs & LSTMs — sequence models before attention

Before transformers, how did a network read a sentence? Not all at once — it read one word at a time, carrying a running memory of everything it had seen so far. That memory is the whole idea behind a recurrent network, and for a decade it was how machines did language.

Why a plain network can’t read a sentence

A feed-forward network takes a fixed-size input. But a sentence can be three words or three hundred, and the order carries the meaning — “dog bites man” is not “man bites dog.” You need something that ingests a sequence step by step and remembers what came before.

Picture how you read: you keep a rough summary in your head and update it after every word. A recurrent neural network (RNN) does exactly that.

The hidden state — one memory, reused

An RNN keeps a vector called the hidden state h. At each step it combines the previous memory with the current word to produce a new memory:

h_t = tanh( W_hh · h_(t-1)  +  W_xh · x_t  +  b )

The same weight matrices W_hh and W_xh are reused at every step — the network learns one update rule and applies it down the whole sequence. The hidden state at the end has, in principle, seen every token that came before it.

The trouble: gradients that vanish or explode

Training an RNN means unrolling it and running backprop down the whole sequence — backprop through time. But that backward pass multiplies by (roughly) the same recurrent matrix once per step. Repeated multiplication is brutal: a factor slightly below 1 shrinks toward zero, and a factor slightly above 1 blows up.

Say the effective per-step factor is 0.8. Over a 50-token sentence the gradient reaching the first word is scaled by:

0.8 ^ 50  ≈  0.0000143

It has essentially vanished — the network cannot learn that the start of the sentence mattered. Nudge the factor to 1.2 and you get the opposite disaster:

1.2 ^ 50  ≈  9100

an exploding gradient that wrecks training. This is the vanishing/exploding gradient problem playing out across time, and it means a plain RNN struggles to connect words more than a few steps apart.

LSTMs — a protected memory highway

The Long Short-Term Memory (LSTM) cell fixes this with a second vector, the cell state C, that runs straight down the sequence like a conveyor belt — edited by addition, not by repeated matrix multiplication. Three learned gates (each a sigmoid that outputs 0–1) decide what happens to it:

• Forget gate — how much of the old cell state to keep.
• Input gate — how much of the new candidate to write in.
• Output gate — how much of the cell state to expose as this step’s hidden state.

Because the cell state is updated additively, gradients can flow back across hundreds of steps without vanishing — the LSTM can remember that a sentence opened with “The keys” long enough to make the verb agree. The GRU is a lighter cousin with just two gates (reset and update) that often works as well for less compute.

Pause and think

An LSTM processes token 100 only after it has processed tokens 1 through 99, in order. What does that force about training on a 10,000-token document — can the work be spread across a GPU’s thousands of cores, or must it happen step by step?

That sequential bottleneck is the answer to why attention won.

Why attention replaced recurrence

Recurrence has two limits that scale poorly:

1. It’s inherently sequential. Step t needs h_(t-1), so you cannot process a sequence in parallel during training — a death sentence on modern GPUs built for thousands of simultaneous operations.
2. The path between distant tokens is long. Information from token 1 to token 500 must survive 499 hops, and even an LSTM strains over very long contexts.

Self-attention removes both: every token looks at every other token directly (a path length of one) and all at once (fully parallel). That is the leap “Attention Is All You Need” made — and why the transformer replaced the RNN as the backbone of modern language models.

In one breath

• An RNN reads a sequence step by step, folding each token into a reused hidden state — the same weights applied at every position.
• Backprop through time multiplies by the recurrent matrix once per step, so gradients vanish (0.8⁵⁰ ≈ 1e-5) or explode (1.2⁵⁰ ≈ 9100) over long sequences.
• The LSTM adds a cell state edited by addition, with forget/input/output gates — a memory highway that survives hundreds of steps; the GRU is a lighter two-gate version.
• LSTMs powered a decade of translation and speech, but they are sequential and slow to train and still strain over very long range.
• Attention replaced them by connecting any two tokens directly and in parallel — the foundation of the transformer.

Check yourself

What to remember

• A recurrent network reads a sequence one step at a time, carrying a hidden state updated by the same weights at every position.
• Backprop through time makes gradients vanish or explode over long sequences, so plain RNNs can’t connect distant words.
• LSTMs (and lighter GRUs) add a gated cell state — a memory highway that survives long range, and the workhorse of pre-2017 sequence modeling.
• Recurrence is sequential and slow; attention connects tokens directly and in parallel, which is why the transformer took over.