Attention (the RNN era)

Seq2seq crammed the entire source into one fixed context vector, and long sentences paid for it. Attention (Bahdanau et al., 2014) is the fix, and it’s one of the most important ideas in deep learning: instead of forcing the decoder to rely on a single summary, let it look back at all the encoder states and, at each step, focus on the ones that matter.

The mechanism: score, softmax, blend

At each decoding step, attention builds a custom context vector just for that step, in three moves. The decoder’s current state is the query; each encoder hidden state is a candidate to attend to.

1. Score — measure how relevant each encoder state is to the query (a dot product, or a small learned function).
2. Softmax — turn the scores into attention weights that sum to 1.
3. Blend — take the weighted sum of the encoder states. That’s the context vector for this step.

import numpy as np

enc_states = {"the": [1, 0], "cat": [0, 1], "sat": [1, 1]}   # one encoder state per source word
query = np.array([0.1, 0.9])                                  # decoder state this step

words = list(enc_states); H = np.array([enc_states[w] for w in words], float)
scores  = H @ query                                          # 1. relevance score per state
weights = np.exp(scores) / np.exp(scores).sum()              # 2. softmax -> attention weights
context = weights @ H                                        # 3. weighted blend

for w, s, a in zip(words, scores, weights):
    print(f"{w:<4} score={s:.2f}  attention={a:.2f}")
print("context vector:", context.round(2))

the  score=0.10  attention=0.18
cat  score=0.90  attention=0.39
sat  score=1.00  attention=0.43
context vector: [0.61 0.82]

The decoder put most of its weight on sat and cat and little on the, and the context is a blend skewed toward them. Next step, with a different query, the weights shift to different words. There is no bottleneck: every encoder state is always reachable, and the decoder decides where to look.

Variants and the alignment bonus

The score function has flavors: Bahdanau (2014) used an additive small neural network; Luong (2015) used the simpler, faster dot product (what the demo uses, and what transformers adopted). Either way, attention came with a bonus: the weights are an interpretable soft alignment — visualize them and you see which source words the model used for each output word (in translation, a near-diagonal map, bending where word order differs). The model learns this alignment without ever being told it.

In one breath

• Attention (Bahdanau, 2014) removes the seq2seq fixed-context bottleneck: instead of one summary vector, the decoder looks back at all encoder states and focuses on the relevant ones.
• The mechanism, per decoding step: score each encoder state against the decoder query, softmax to attention weights, blend (weighted sum) into a step-specific context vector (the demo weights sat/cat over the).
• There is no bottleneck — every encoder state is always reachable, and the weights change each step.
• Bahdanau used additive scoring, Luong the faster dot-product; the weights double as an interpretable soft alignment (which source words map to each output word), learned automatically.
• This is cross-attention on an RNN; dropping the RNN and attending within a sequence (self-attention) is the transformer — attention became the foundation of modern NLP.

Quick check

Next

Attention taken to its conclusion — dropping the RNN, attending within one sequence — is self-attention and the transformer. It grew out of the seq2seq bottleneck it was built to fix.