What is the difference between batch normalization and layer normalization, and why do transformers use layer norm?

How to think about it

Batch normalization and layer normalization both center and scale activations, but they average over different axes. Batch norm uses other examples in the minibatch for each feature and normally switches to stored statistics at inference; layer norm uses the features of one example, usually one token’s hidden vector, and computes its statistics independently. Transformers prefer layer norm because their batches can be small or changing, their sequences have different lengths and padding, and a token should not change merely because an unrelated request happened to share its batch.

Why the two normalizations differ

An activation is the intermediate numeric output produced inside a neural network. A feature is one coordinate of that output. Normalization keeps these coordinates on a manageable scale, which makes optimization less sensitive to changing activation magnitudes.

Imagine a minibatch as a matrix with B examples and D features. Batch normalization computes one mean and variance for each feature, using the B examples:

mu[j]  = average(x[i][j] for i in the batch)
var[j] = average((x[i][j] - mu[j]) ** 2 for i in the batch)

y[i][j] = gamma[j] * (x[i][j] - mu[j]) / sqrt(var[j] + epsilon) + beta[j]

Here, epsilon is a small positive number that prevents division by zero. gamma and beta are learned scale and shift parameters. They matter because normalization should not permanently force every feature to have mean zero and unit variance; the model can learn to undo or adjust the transformation when useful.

During training, batch norm uses the current minibatch’s statistics. During inference, frameworks normally use running estimates collected during training instead. That difference is important. Training with one group of examples and serving with another can expose a mismatch in activation distributions.

Layer normalization changes the axis. It takes one example’s feature vector and computes its mean and variance across the D features:

mu[i]  = average(x[i][j] for j in the features)
var[i] = average((x[i][j] - mu[i]) ** 2 for j in the features)

y[i][j] = gamma[j] * (x[i][j] - mu[i]) / sqrt(var[i] + epsilon) + beta[j]

For a transformer, “one example” usually means one token position. If the hidden size is 768, layer norm uses the 768 numbers belonging to that token. It does not use the other examples in the batch and, in standard use, it does not use the other tokens in the sequence.

Common misconception: “Layer” in layer normalization does not mean that the normalization averages across all layers of the network. Standard LayerNorm(d_model) normalizes the final hidden dimension for each token independently.

Layer norm therefore has no running mean or running variance. The same kind of calculation happens during training and inference. Other parts of the model, such as dropout, can still behave differently between those modes, but layer norm itself does not create a train-versus-evaluation statistics switch.

A four-number example

Take two four-feature vectors:

• Example A: [2, 4, 6, 8]
• Example B: [10, 12, 14, 16]

Assume the learned scale is one, the learned shift is zero, and use population variance.

For layer norm, Example A has mean 5 and variance 5. Its normalized output is approximately:

[-1.342, -0.447, 0.447, 1.342]

Example B has mean 13 and the same variance 5, so its normalized output is exactly the same pattern:

[-1.342, -0.447, 0.447, 1.342]

Layer norm has removed the absolute offset of each vector while preserving its internal shape.

For batch norm, look at each feature column across the two examples. The first feature is [2, 10], with mean 6 and variance 16, so it becomes [-1, 1]. The second feature is [4, 12], with mean 8, and also becomes [-1, 1]. The same happens for every column. The outputs are therefore:

• Example A: [-1, -1, -1, -1]
• Example B: [1, 1, 1, 1]

That is the central distinction. Layer norm asks, “How are this example’s features distributed relative to one another?” Batch norm asks, “How does this example’s value for each feature compare with the other examples in the batch?”

If Example A were sent through the model beside a completely different group of examples, batch norm’s result could change. Layer norm’s result would not, assuming the token’s own hidden vector stayed the same.

Why transformers favor layer norm

A transformer commonly works with a tensor shaped like B × T × D: batch size, number of token positions, and hidden size. Self-attention lets tokens exchange information with other tokens in the same sequence. That interaction is intentional. Batch norm would introduce another interaction across unrelated sequences.

Suppose a serving batch contains 32 ordinary customer questions and one 2,000-token legal document. If batch norm is applied across token positions, the long document contributes many more values to the feature statistics. The representations of the short questions can shift because of a document they have nothing to do with. That is a poor property for a request-serving system.

Variable sequence lengths make this worse. Hardware prefers rectangular tensors, so shorter sequences are padded. Attention masks stop attention from using padding as meaningful content, but a standard batch-normalization operation does not automatically understand that mask. Padding values can still influence its mean and variance. Layer norm avoids this particular problem because the statistics for a real token come only from that token’s hidden features.

Autoregressive generation makes batch independence even more valuable. During decoding, a system may serve one request at a time, so the batch size is one, or continuously change the batch as requests arrive and finish. Layer norm still has all D features of the current token available. Batch norm has no useful collection of unrelated examples to estimate statistics from. It can use frozen training statistics, but those statistics may not match the serving distribution.

Transformers also contain residual connections. A residual connection adds a sublayer’s output back to its input, as in x + sublayer(x). Repeating that operation dozens or hundreds of times can make activation scales difficult to control. Layer norm provides a stable scale for each token before or after attention and the feed-forward network, while its learned gamma and beta parameters preserve flexibility.

A common pre-layer-normalized block looks conceptually like this:

h1 = h + Attention(LayerNorm(h))
h2 = h1 + FeedForward(LayerNorm(h1))

The original Transformer used post-layer normalization, where the residual addition is normalized afterward. The choice between pre-LN and post-LN is separate from the choice between batch and layer normalization. Many deep transformer implementations favor pre-LN because the residual stream provides a cleaner direct path for gradients, which often makes very deep models easier to optimize.

The senior-level nuance

“Transformers use layer norm” is a useful default answer, not a universal law.

Batch norm can work if the architecture has large, stable batches and carefully handles sequence dimensions and padding. Specialized transformer variants have used batch-style normalization. In distributed training, however, batch norm may require synchronized statistics across devices. Without synchronization, two workers can normalize the same feature using different local batches. Layer norm needs no such cross-device coordination.

Batch norm also has a useful side effect: variation in minibatch statistics can act as a form of regularization. That can help convolutional vision models, where batch sizes and spatial structure are often favorable to batch norm. Layer norm does not provide that same batch-based noise. It is chosen for transformers mainly because its independence is valuable, not because it is always a better normalization method.

Many modern language models use RMSNorm instead. RMSNorm scales a vector using its root-mean-square magnitude but does not subtract the mean. It is a close relative of layer norm, with a slightly different trade-off. A precise answer is therefore: transformers typically use layer norm or a layer-normalization alternative such as RMSNorm, rather than batch norm.

A failure mode you can recognize in production

A classic failure is training with batch norm on padded sequences and then serving with different batch sizes or length distributions. Offline validation may look fine, while production accuracy drops at batch size one. If the model is accidentally left in training mode, identical inputs can also produce different logits depending on which other requests share the batch.

Another implementation mistake is normalizing across both sequence length and hidden size when only the hidden size should be normalized. The first symptom is that a token’s output changes when extra padding is added, or when the same prompt is evaluated at a different maximum sequence length. In a causal model, including future positions in such statistics can also introduce information leakage. The safe standard is to normalize each token over its hidden dimension and keep attention masking separate.

What they’ll ask next

Does layer norm use running statistics?

No. It computes the mean and variance from the current token’s feature vector on every forward pass. That is why layer norm itself behaves consistently across training and inference. Dropout and other layers can still differ between those modes.

Can batch norm work with a batch size of one?

For a dense feature vector, it has no useful cross-example statistics during training. In convolutional settings, spatial positions can sometimes provide additional values, so “batch size one” is not automatically impossible there. At inference, batch norm can use frozen running statistics, but those statistics must represent the serving distribution.

Is pre-LN always better than post-LN?

No. Pre-LN often makes deep transformer optimization easier because the residual path carries gradients more directly. Post-LN can still work and was used in the original Transformer, but it can be more sensitive to initialization, learning-rate schedules, and depth. The right choice depends on the architecture and training recipe.

Say this in the interview

Batch norm normalizes each feature across the minibatch and therefore depends on batch composition and running inference statistics; layer norm normalizes each token across its own hidden features, so it remains stable with variable lengths, padding, distributed workers, and batch size one, which is why it is the standard transformer choice.