What is RMSNorm and why do modern LLMs like Llama use it instead of LayerNorm?

For ML Engineer research-engineer AI / LLM Engineer

The short answer

RMSNorm normalizes activations by their root-mean-square only, dropping the mean-centering and bias terms used in LayerNorm. It is roughly 10 to 20 percent cheaper with no measurable quality loss on language modeling, which is why Llama, Mistral, and most modern open-weight LLMs adopt it.

How to think about it

RMSNorm normalizes activations by their root-mean-square only, dropping the mean-centering and bias terms used in LayerNorm. It is roughly 10 to 20 percent cheaper with no measurable quality loss on language modeling, which is why Llama, Mistral, and most modern open-weight LLMs adopt it.

Learn it properly Inside the transformer block

Keep practising

How does LoRA work and why is it preferred over full fine-tuning for large models? How does LLMOps differ from classical MLOps, and what new operational challenges do LLMs introduce? How do SGD, SGD with momentum, and RMSProp differ, and what does each one fix? What is the difference between batch normalization and layer normalization, and why do transformers use layer norm? How would you reduce the cost of serving an ML or LLM model in production without hurting quality?

All Deep Learning questions

Explore further

LLMOps — operating LLMs Dropout, BN, LN What an LLM is

Layer Normalization Batch Normalization L1 Regularization (Lasso) LoRA