What is RMSNorm and why do modern LLMs like Llama use it instead of LayerNorm?

How to think about it

RMSNorm is a normalization layer that divides each token’s hidden vector by its root-mean-square, while LayerNorm first subtracts the vector’s mean and then divides by its standard deviation. Llama uses RMSNorm because mean-centering is often unnecessary in a pre-norm transformer, and removing it makes the operation simpler and sometimes faster without sacrificing useful model quality.

The mechanism

A transformer activation is the vector carried by one token at one layer. If the hidden size is d, that vector has d numbers. In a Llama-style model with hidden size 8,192, every token is represented by an 8,192-dimensional vector.

Normalization keeps the magnitude of that vector under control. This matters because every transformer block repeatedly adds residuals, applies linear projections, and passes values through attention and feed-forward networks. Without some control, activation scales can drift during training, making gradients harder to optimize.

LayerNorm computes a mean and a variance for each token’s hidden vector:

μ = mean(x)

σ² = mean((x − μ)²)

It then produces:

LayerNorm(x) = γ ⊙ (x − μ) / sqrt(σ² + ε) + β

Here, γ is a learned per-coordinate scale, β is a learned per-coordinate offset, and ε is a small positive number that prevents division by zero.

RMSNorm skips the mean and variance calculation. It computes only the root-mean-square:

r = sqrt(mean(x²) + ε)

Then it produces:

RMSNorm(x) = γ ⊙ x / r

The standard RMSNorm version has the learned scale γ but no learned bias β.

Property	LayerNorm	RMSNorm
Centers the vector	Yes	No
Controls magnitude	Yes	Yes
Learned parameters	Scale and usually bias	Usually scale only
Shift-invariant	Yes	No
Runtime cost	Generally higher	Generally lower

The important distinction is not “one normalizes and the other does not.” Both normalize. RMSNorm normalizes the vector’s energy, while LayerNorm normalizes its spread around the mean.

A numerical example

Take a three-dimensional activation:

x = [1, 2, 3]

Assume the learned scale is all ones and ignore the tiny epsilon for readability.

LayerNorm finds a mean of 2. Its centered vector is:

[-1, 0, 1]

The variance is 2/3, so the standard deviation is approximately 0.816. The result is approximately:

[-1.225, 0, 1.225]

RMSNorm does not subtract the mean. It computes:

sqrt((1² + 2² + 3²) / 3) = sqrt(14/3) ≈ 2.160

The result is therefore approximately:

[0.463, 0.926, 1.389]

Both outputs have controlled magnitude. LayerNorm has forced the average value to zero. RMSNorm has not.

That difference explains a common misconception.

The common wrong answer is that RMSNorm also removes the mean. It does not. For example, with x = [10, 10, 10], LayerNorm produces three zeros when its bias is zero, because there is no variation around the mean. RMSNorm produces approximately [1, 1, 1], because the vector has nonzero magnitude. RMSNorm treats a uniform offset as part of the representation instead of automatically declaring it irrelevant.

Why this works in LLMs

Modern decoder-only LLMs commonly use a pre-norm structure. In simplified form, a transformer block looks like this:

h = h + attention(rms_norm(h))
h = h + mlp(rms_norm(h))

The normalization controls the input to attention and the feed-forward network, while the residual stream h still has a direct path through the block. That direct path is valuable for optimization: information and gradients can travel through repeated additions instead of passing through every nonlinear sublayer.

In this setting, subtracting the mean is not always buying enough to justify its cost. The model can learn useful representations with nonzero offsets, and the residual architecture already provides a stable route for information. RMSNorm still removes sensitivity to the overall scale of a vector. If the vector is multiplied by two, both its numerator and RMS are approximately multiplied by two, so the normalized direction stays nearly the same.

That is the practical argument. It is not a theorem saying that mean-centering never matters. It is an architectural choice supported by training results: for many language-modeling workloads, RMSNorm gives comparable quality while doing less work.

The quality result belongs to the whole training recipe. Norm placement, initialization, learning rate, epsilon, precision, model size, and data all matter. RMSNorm is not a magic replacement that improves every model.

Why it can be cheaper

LayerNorm must:

1. Reduce the vector to find its mean.
2. Subtract that mean from every coordinate.
3. Compute the variance.
4. Take a square root and divide.
5. Apply a learned scale and usually a learned bias.

RMSNorm still needs a reduction, a square root, and a scale operation. It is not half the work. But it removes the mean calculation, the centering pass, and the bias addition. A fused GPU kernel can also avoid storing and rereading some intermediate values.

The parameter saving is tiny. Removing one bias vector saves only d parameters per normalization layer. The more meaningful benefit is execution: fewer operations and less memory traffic in a function called repeatedly across every layer and token.

Consider a Llama-style 70B decoder with 80 layers and hidden size 8,192. If each layer has two normalization points, that is approximately:

80 × 2 × 8,192 = 1,310,720

normalized scalar positions for every generated token, before counting the final normalization. Over 100 generated tokens, the model processes about 131 million such positions. Simplifying each normalization can matter, particularly during token-by-token decoding, where small kernels and memory movement become more visible.

However, I would not promise an automatic 10 to 20 percent end-to-end speedup. The actual result depends on the kernel implementation, GPU, batch size, sequence length, precision, and whether matrix multiplications or memory access dominate the workload. Prompt processing is often dominated by large matrix operations. Decoding can expose normalization overhead more clearly. The right answer is to benchmark p50 and p99 latency on the actual serving setup.

The production pattern

I would choose RMSNorm when training a compatible architecture from scratch or when the model specification already calls for it. I would record the normalization type, epsilon, placement, and numerical precision as part of the checkpoint configuration.

An illustrative PyTorch-style implementation is:

def rms_norm(x, weight, eps=1e-6):
    mean_square = x.float().pow(2).mean(dim=-1, keepdim=True)
    scale = (mean_square + eps).rsqrt()
    y = x.float() * scale * weight.float()
    return y.to(x.dtype)

The reduction is over the last dimension, which is the hidden dimension. The exact epsilon should come from the model configuration; 1e-6 here is only an illustration. Implementations commonly accumulate the sum of squares in float32 even when the model uses float16 or bfloat16, then cast the result back.

I would not replace LayerNorm with RMSNorm in a pretrained checkpoint and assume it is an inference optimization. The surrounding weights were trained for a particular activation distribution. Changing the normalization changes that distribution immediately. The usual symptom is a large validation-loss jump or noticeably degraded, repetitive generations. The model generally needs to be retrained or carefully adapted.

I also would not choose RMSNorm solely because it is fashionable. If the target hardware has a highly optimized LayerNorm kernel and no efficient RMSNorm kernel, the measured latency difference may be negligible. If experiments show that mean-centering helps a particular architecture or task, LayerNorm is the correct choice. The model’s benchmark decides, not the acronym.

What they’ll ask next

Does RMSNorm remove the mean?
No. It divides by the root-mean-square but leaves the mean generally nonzero. LayerNorm is invariant to adding the same constant to every coordinate; RMSNorm is not.

Is RMSNorm always faster than LayerNorm?
Usually it has less mathematical and memory work, but there is no universal speedup. Fused kernels, hardware, batch size, sequence length, and the rest of the model determine end-to-end latency.

Can I swap LayerNorm for RMSNorm without retraining?
Generally no. It changes the inputs seen by every attention and feed-forward sublayer. Treat it as an architectural change unless you have a deliberate adaptation procedure and evaluation showing compatibility.

Why not remove normalization altogether?
Normalization constrains activation scale before the sublayers, which makes optimization more stable. RMSNorm removes one part of LayerNorm while retaining that scale control.

Say this in the interview

“RMSNorm controls a token vector’s magnitude using its root-mean-square but does not subtract its mean; Llama uses it in a pre-norm transformer because mean-centering is often unnecessary there, and removing it gives a simpler, potentially faster kernel with comparable quality, although the real speedup and quality still depend on the implementation and training recipe.”