Dropout, BatchNorm, LayerNorm

Three techniques show up in almost every modern deep network: dropout, batch normalization, and layer normalization. They solve different problems but are easy to confuse. Here’s what each one actually does and when to use it.

Dropout — averaging over subnetworks

At training time, dropout randomly zeros out a fraction p of activations (say 10–50%). At test time (inference), it does nothing — all neurons are active. A common mistake is forgetting model.eval(), which leaves dropout on and makes predictions non-deterministic.

import numpy as np

rng = np.random.default_rng(42)

def dropout(x, p, training=True):
    if not training or p == 0:
        return x
    # Inverted dropout: scale by 1/(1-p) so expected magnitude is unchanged
    mask = (rng.random(x.shape) > p).astype(x.dtype)
    return x * mask / (1 - p)

x = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0])
print("input:        ", x)
print("dropout p=0.5:", dropout(x, 0.5, training=True))
print("dropout p=0.5:", dropout(x, 0.5, training=True))   # different mask each call
print("eval mode:    ", dropout(x, 0.5, training=False))

input:         [1. 2. 3. 4. 5. 6. 7. 8.]
dropout p=0.5: [ 2.  0.  6.  8.  0. 12. 14. 16.]
dropout p=0.5: [ 0.  0.  0.  8. 10. 12.  0.  0.]
eval mode:     [1. 2. 3. 4. 5. 6. 7. 8.]

Two things to notice:

1. The mask changes every call. Each training step sees a different “thinned” network. You’re effectively training an ensemble of exponentially many subnetworks, all sharing weights.
2. Survivors get scaled up by 1/(1-p). This keeps the expected activation magnitude the same as without dropout — so you don’t need to change anything at inference time.

Why it regularizes: a feature can’t rely on any single co-adapted upstream feature (it might be dropped). The network spreads useful work across many features — more robust representations.

Batch normalization — normalize per minibatch

BatchNorm normalizes each feature channel across the batch dimension: subtract the batch mean, divide by the batch std, then apply a learnable scale + shift.

y = γ · (x − μ_batch) / sqrt(σ²_batch + ε) + β

It was the breakthrough that made deep CNN training stable in 2015. Two big advantages:

• Activations stay roughly mean-0, variance-1 throughout the network.
• You can use much higher learning rates without divergence.

Two big downsides:

• Different behavior in train vs eval mode. During training, you normalize with batch statistics. At eval, you use running averages accumulated during training. This means batch size affects your results.
• Awful for small batches. With batch size 1 or 2, batch statistics are noisy. With batch size 8, they’re still iffy.

BatchNorm is still dominant in vision CNNs (ResNet, ConvNeXt). It’s mostly out of fashion in transformers and small-batch training.

Layer normalization — normalize per sample

LayerNorm normalizes each sample across the feature dimension:

y = γ · (x − μ_features) / sqrt(σ²_features + ε) + β

The key difference: no dependence on other samples in the batch. Same behavior at train and eval. Works with batch size 1. Doesn’t care if your minibatch is small.

import numpy as np

# (batch, features) = (3, 4)
x = np.array([
    [1.0, 2.0, 3.0, 4.0],
    [10.0, 0.0, 5.0, 5.0],
    [-1.0, -2.0, -3.0, -4.0],
])

# LayerNorm: normalize each ROW (across feature axis)
mean = x.mean(axis=-1, keepdims=True)
std = x.std(axis=-1, keepdims=True) + 1e-5
layer_norm = (x - mean) / std

# BatchNorm: normalize each COLUMN (across batch axis)
mean_b = x.mean(axis=0, keepdims=True)
std_b = x.std(axis=0, keepdims=True) + 1e-5
batch_norm = (x - mean_b) / std_b

print("LayerNorm (each row mean≈0, std≈1):")
print(layer_norm.round(2))
print("BatchNorm (each col mean≈0, std≈1):")
print(batch_norm.round(2))

LayerNorm (each row mean≈0, std≈1):
[[-1.34 -0.45  0.45  1.34]
 [ 1.41 -1.41  0.    0.  ]
 [ 1.34  0.45 -0.45 -1.34]]
BatchNorm (each col mean≈0, std≈1):
[[-0.49  1.22  0.39  0.58]
 [ 1.39  0.    0.98  0.83]
 [-0.91 -1.22 -1.37 -1.41]]

Same operation, different axis. The axis choice changes everything:

	BatchNorm	LayerNorm
Normalizes across	batch axis	feature axis
Train ≠ Eval?	yes (uses running stats)	no
Works with batch=1?	no	yes
Sensitive to batch size?	yes	no
Used in	CNNs	Transformers, RNNs

RMSNorm — the modern variant

Modern LLMs increasingly use RMSNorm — LayerNorm without the mean subtraction:

y = γ · x / sqrt(mean(x²) + ε)

Slightly cheaper (no mean, no bias), empirically just as good — and now the default in essentially every open LLM (Llama, Mistral, Qwen, Gemma, DeepSeek). Where it sits relative to the residual connection (pre-norm) is covered in inside the transformer block.

Putting it in PyTorch

# CNN: BatchNorm after conv, before activation
nn.Sequential(
    nn.Conv2d(64, 128, 3, padding=1),
    nn.BatchNorm2d(128),
    nn.ReLU(),
    nn.Dropout(0.1),
)

# Transformer block: LayerNorm in pre-norm position (modern standard)
nn.Sequential(
    nn.LayerNorm(d_model),
    nn.Linear(d_model, d_ff),
    nn.GELU(),
    nn.Linear(d_ff, d_model),
)

model.train() and model.eval() flip dropout and batch-norm’s behavior. Forgetting to call model.eval() before inference is a classic bug — predictions become non-deterministic and BatchNorm uses the wrong statistics.

In one breath

• Dropout randomly zeros a fraction p of activations in training (off at inference), scaling survivors by 1/(1−p) — it trains an ensemble of subnetworks so no feature can lean on a single co-adapted neighbor.
• It’s standard for fine-tuning and smaller models but often near-zero in large LLM pretraining, which is data-bottlenecked, not overfitting.
• BatchNorm normalizes each feature across the batch — great for vision CNNs and high learning rates — but differs train vs eval and breaks on tiny batches.
• LayerNorm normalizes each sample across its features — no batch dependence, same in train and eval, works at batch size 1 — which is why transformers use it.
• Modern LLMs use RMSNorm (LayerNorm minus the mean-subtraction): cheaper, just as good. Always call model.eval() before inference, or dropout and BatchNorm misbehave.

Quick check

Quick check

0/2

Q1You run inference with `model.train()` left on. What goes wrong?

Nothing — train/eval only affect trainingDropout zeros random activations, and BatchNorm uses noisy batch stats — predictions become non-deterministicPyTorch raises an errorGradients get computed wastefully

Check answer

Q2Why do transformers use LayerNorm instead of BatchNorm?

LayerNorm is cheaper to computeLayerNorm doesn't depend on the batch dimension, so it handles variable-length sequences and small batches gracefullyBatchNorm doesn't have learnable parametersLayerNorm provides better regularization

Check answer