The Transformer Architecture

This is the overview that ties the chapter together. You have already met the pieces in detail — self-attention (the core mechanism), multi-head attention, and positional encodings.

Here we assemble them into the whole machine and see the shape of the thing.

The headline from the paper: the Transformer is “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.” No RNNs (models that read one word at a time), no convolutions. What remains is attention, a couple of MLPs (small feed-forward networks), and a stack of residual connections (shortcuts that add a layer’s input to its output). That simplicity is exactly why it scaled.

The two halves

The model has a left half and a right half. The encoder reads the entire input at once and turns it into a stack of context-rich vectors.

The decoder generates the output one token at a time. It looks both at what it has produced so far and — through cross-attention — at the encoder’s output.

Both halves stack N identical layers — the paper used N = 6. “Identical” means identical in structure; each layer has its own independent learned weights.

Every sub-layer and embedding outputs vectors of the same width d_model (512 in the paper) so that residual connections line up.

The encoder layer, exactly

Each encoder layer is two sub-layers, in this order:

1. Multi-Head Self-Attention then Add & Norm
2. Position-wise Feed-Forward then Add & Norm

That is the whole recipe. Attention mixes information across tokens; the feed-forward network then transforms each position on its own.

Each sub-layer is independently wrapped in a residual connection followed by layer normalization — the paper’s formula is LayerNorm(x + Sublayer(x)).

The decoder layer adds one more sub-layer in the middle:

1. Masked Multi-Head Self-Attention then Add & Norm
2. Multi-Head Cross-Attention then Add & Norm
3. Position-wise Feed-Forward then Add & Norm

The order matters, and a lot of mental models get it wrong. So here is the canonical stack, drawn bottom-up the way data actually flows:

Why each block is there

Strip away the names and every block earns its place by solving one concrete problem:

• Embedding — attention does linear algebra on vectors, not on symbols. The embedding maps each token id to a learned d_model-vector so the vocabulary becomes geometry.
• Positional Encoding — self-attention is permutation-invariant: shuffle the tokens and the math is unchanged. Without position information the model sees a bag of words. So a position-dependent signal is added (not concatenated — that is why it shares d_model) to each embedding.
• Multi-Head Self-Attention — lets every token pull in information from every other token in one shot. Several heads run in parallel so the model can attend in different representation subspaces at once.
• Add & Norm — the residual connection lets gradients flow straight through a deep stack; layer normalization keeps activations well-scaled. Together they are what make stacking N layers actually trainable.
• Feed-Forward — attention mixes tokens together; the position-wise MLP (two linear layers with a ReLU between) then transforms each position non-linearly on its own.
• Masked Self-Attention — in the decoder, future positions are blocked so a token can only depend on earlier ones. Without it the model would cheat during training by reading the answer.
• Cross-Attention — the decoder’s queries read the encoder’s keys and values, so the output can condition on the encoded input. This is the only place the two halves talk.
• Linear then Softmax — at the very top, a Linear layer projects the final decoder vector to one score per vocabulary word. Softmax then turns those scores into a probability distribution over the next token.

Masked self-attention is not cross-attention

This is the distinction the diagram is built to drill in, because it is the one people get wrong most often.

Same attention math, completely different sources for the keys and values.

The masked self-attention layer has a causal mask (a filter that hides later tokens) so a token never sees the future. The cross-attention layer has no causal mask at all — the decoder is allowed to look at the entire input sequence, because the input is already fully known.

The output layer: Linear, then Softmax

Order matters here too. The decoder’s final hidden vector is d_model-wide.

A Linear layer projects it up to the size of the vocabulary, producing one raw score — a logit — per possible next token. Softmax then exponentiates and normalizes those logits into a probability distribution. Linear first, softmax second. (More on the output layer in the softmax lesson.)

Softmax itself is three lines of NumPy — exponentiate, then divide by the sum, with a max-subtraction for numerical stability:

import numpy as np

def softmax(z):
    z = z - z.max()          # stability: shift so the largest logit is 0
    e = np.exp(z)
    return e / e.sum()       # now sums to exactly 1

# Pretend these are the model's logits over a tiny 5-word vocabulary.
logits = np.array([2.1, 0.4, -1.2, 3.6, 0.9])
probs = softmax(logits)

print("probabilities:", np.round(probs, 4))
print("sum:", probs.sum())          # 1.0
print("argmax (greedy next token):", probs.argmax())

probabilities: [0.1666 0.0304 0.0061 0.7466 0.0502]
sum: 0.9999999999999999
argmax (greedy next token): 3

Every output lies in (0, 1) and they sum to 1. You can sample from the distribution or just take the argmax for a greedy pick.

The three families — same blocks, rearranged

Here is the aha the architecture is built around. The encoder, the decoder, and the output head are lego pieces.

Snap them together differently and you get three different kinds of model. The full lesson on choosing between them is BERT, GPT, T5 — here is the structural summary.

The decoder-only column is worth dwelling on. A decoder-only model drops the encoder and the cross-attention sub-layer entirely. There is no encoder output to attend to.

Each block is just masked self-attention plus a feed-forward network (with their norms). That is GPT, Llama, and Claude.

Why modern LLMs are decoder-only

If the original was an encoder-decoder, why did the field consolidate on decoder-only? A few reasons:

• One objective that scales. Predicting the next token is a single, simple training signal that works on essentially any text, and it scales beautifully with data and parameters.
• The prompt replaces the encoder. Anything you would have fed the encoder, you can just put in the prompt as context the model attends to. So the encoder is not strictly necessary.
• Architectural simplicity. No separate encoder, no cross-attention — fewer moving parts to scale and serve.
• It worked. In 2020 to 2023, the generative quality of GPT-3, ChatGPT, and GPT-4 settled the argument in practice. Decoder-only became the default for general-purpose LLMs, and newer models have kept it.

Encoder-only models (BERT and its descendants) are still the backbone of classification, retrieval, and embeddings. Encoder-decoder models (T5, BART) still win on explicit sequence-to-sequence tasks.

But for the chat-style models this site is mostly about — see what is an LLM — decoder-only is the shape.

In one breath

• The 2017 Transformer is an encoder–decoder: the encoder reads the whole input bidirectionally into context vectors; the decoder generates one token at a time, masked so it can’t peek ahead.
• Each layer is the same few lego pieces — embedding, positional encoding, multi-head attention, Add & Norm (residual + normalization), feed-forward — stacked N times at a constant width so residuals line up.
• The one bridge between halves is cross-attention: the decoder’s queries read the encoder’s keys and values. That is not the decoder’s lower masked self-attention (Q, K, V all from the decoder). Conflating the two is the classic mistake.
• At the top, Linear then Softmax turns the final vector into a probability over the vocabulary.
• Rearrange the same blocks and you get three families: encoder-only (BERT, understanding), encoder-decoder (T5, seq-to-seq), and decoder-only (GPT/Llama/Claude, which drop the encoder and cross-attention).
• Decoder-only won for general LLMs because next-token prediction scales and the prompt replaces the encoder.

Quick check

Next

You now have the whole machine in view.

To go deeper on the pieces, revisit self-attention for the Q·K·V math, multi-head attention for the parallel subspaces, and positional encodings for how order gets injected.

To go up a level into how these models are used in practice, head to what is an LLM.