Anatomy of a modern LLM

The 2017 “Attention Is All You Need” transformer is still the skeleton of every LLM — but almost every organ has been swapped. If you opened LLaMA, Mistral, or Qwen3 expecting the original block, you’d find five quiet upgrades. This lesson walks the modern decoder block and explains each change on its own terms.

What actually changed

Everything else — residual connections, the attention-then-FFN rhythm, stacking L identical blocks — is unchanged. Let’s take the new organs one at a time.

RMSNorm, and why pre-norm

The original block applied LayerNorm after the residual add (post-norm), which makes very deep stacks hard to train. Modern LLMs put the norm before each sublayer (pre-norm) so a clean residual highway runs uninterrupted from input to output — gradients flow, and 100-layer models train stably.

They also drop LayerNorm for RMSNorm, which skips the mean-centering and just rescales by the root-mean-square. Fewer operations, no bias term, and in practice no quality loss:

import numpy as np
x = np.array([2.0, -1.0, 0.5, 3.0, -2.0])

# LayerNorm: subtract mean, divide by std, then scale/shift
ln = (x - x.mean()) / (x.std() + 1e-5)

# RMSNorm: no mean-centering — just divide by the root-mean-square
rms = x / (np.sqrt(np.mean(x**2)) + 1e-5)

print("LayerNorm:", np.round(ln, 3))
print("RMSNorm  :", np.round(rms, 3))
# RMSNorm drops the mean subtraction and the bias — fewer ops, same stability.

LayerNorm: [ 0.813 -0.813  0.     1.356 -1.356]
RMSNorm  : [ 1.047 -0.523  0.262  1.57  -1.047]

LayerNorm re-centers around zero (note its output averages to 0); RMSNorm skips that step and only rescales — fewer operations, no bias, and in practice no quality loss.

RoPE — position without a position embedding

The original added a fixed positional vector to each token. Modern models use rotary positional embeddings (RoPE): they rotate the query and key vectors by an angle proportional to position before the dot product. Because attention then depends on the difference of rotation angles, the model sees relative position for free — and you can stretch RoPE to context lengths it never trained on. (Full derivation in Positional encodings & RoPE.)

GQA — the same attention, a smaller cache

Multi-head attention gives every query head its own key and value heads. At inference you must cache K and V for every token, and that KV cache is usually the memory bottleneck. Grouped-query attention lets several query heads share one K/V head — e.g. 32 query heads but only 8 K/V heads cuts the cache 4×, with negligible quality loss. (Details and the MHA→MQA→GQA→MLA ladder in Multi-head attention.)

SwiGLU — the gated feed-forward

The biggest change inside the block is the FFN. The original was Linear → ReLU → Linear. Modern models use SwiGLU: split the input into two linear paths — a value path and a gate path — pass the gate through SiLU (x·σ(x)), then multiply them elementwise before projecting down:

SwiGLU(x) = ( SiLU(x · W_gate) ⊙ (x · W_up) ) · W_down

Why is a multiply better than a ReLU? A ReLU can only switch a feature fully on or off. The gate is a smooth, learned, per-feature volume knob — the network decides how much of each feature to let through. Where SiLU(gate) ≈ 0 the feature is suppressed no matter how large its value; where the gate opens, the feature passes and can even be amplified.

Run four features through the gate and watch the volume knob act independently on each:

import numpy as np
def silu(z): return z / (1.0 + np.exp(-z))   # SiLU(z) = z·σ(z)

value = np.array([ 1.8, -2.0,  0.9,  3.1])   # the up path    (x·W_up)
gate  = np.array([ 2.5, -3.0,  0.1,  1.2])   # the gate path  (x·W_gate)
out   = value * silu(gate)                    # elementwise gated value

print("value      :", np.round(value, 2))
print("SiLU(gate) :", np.round(silu(gate), 3))
print("gated out  :", np.round(out, 3))

value      : [ 1.8 -2.   0.9  3.1]
SiLU(gate) : [ 2.31  -0.142  0.052  0.922]
gated out  : [4.159 0.285 0.047 2.859]

Feature index 2 has a healthy value of 0.9, but its gate is nearly shut (SiLU(0.1) ≈ 0.05), so it leaves as 0.047 — effectively switched off. Feature 0’s open gate (SiLU(2.5) ≈ 2.31) amplifies it from 1.8 to 4.16. That independent, per-feature control is exactly what a single ReLU cannot do.

(To keep the parameter count fair, SwiGLU uses a smaller hidden width — about ⅔ — since it has three weight matrices instead of two.)

Putting the block together

Stack it and you have a modern decoder layer — L of these, then a final RMSNorm and the output projection:

x ─►┌─ RMSNorm ─► GQA attention (RoPE on Q,K) ─┐─► + ─►┌─ RMSNorm ─► SwiGLU FFN ─┐─► + ─►
    └──────────── residual ───────────────────┘       └──────── residual ───────┘

In one breath

• A modern LLM is the 2017 transformer with five organs swapped; the residual + attention-then-FFN skeleton is untouched.
• RMSNorm + pre-norm: drop the mean-centering and put the norm inside the residual branch, so the highway stays clean and deep stacks train stably.
• RoPE rotates Q and K for relative position that extrapolates to longer context; GQA lets several query heads share one K/V head, shrinking the inference KV cache (e.g. 32 query / 8 K-V heads = 4× smaller).
• SwiGLU replaces the ReLU MLP with SiLU(x·W_gate) ⊙ (x·W_up) then ·W_down — a smooth per-feature volume knob (a near-shut gate kills a feature whatever its value; an open gate amplifies). MoE optionally swaps the dense FFN for many experts with a router, scaling parameters without scaling per-token compute — the dense-vs-MoE split in Qwen3.

Check yourself

What to remember

• A modern LLM is the 2017 transformer with five organs swapped: RMSNorm + pre-norm, RoPE, GQA, SwiGLU, and optionally MoE.
• RMSNorm drops mean-centering (cheaper); pre-norm keeps the residual highway clean for stable deep training.
• RoPE rotates Q/K for relative position that extrapolates; GQA shares K/V heads to shrink the inference KV cache.
• SwiGLU multiplies a value path by a learned SiLU gate — a smooth per-feature volume knob that beats a ReLU MLP. MoE scales parameters without scaling per-token compute.