What is the difference between Xavier (Glorot) and He initialization, and when do you use each?

How to think about it

Use Xavier, also called Glorot initialization, for linear, tanh, and usually sigmoid layers; use He, also called Kaiming initialization, for ReLU-family layers such as ReLU, LeakyReLU, and PReLU. Xavier uses both fan-in and fan-out to balance signal flow, while He uses a larger forward-preserving scale because ReLU discards roughly half of a zero-centred signal.

Why the choice matters

Imagine a 30-layer multilayer perceptron at random initialization. Every layer receives numbers from the previous layer and sends numbers to the next one. If the weights are too large, activations and gradients grow as they pass through the network. If they are too small, both shrink toward zero. The model may still run, print plausible loss values, and learn almost nothing. That is a particularly expensive way to discover a square-root factor.

For a layer, fan-in means the number of inputs feeding one output unit. Fan-out means the number of output units affected by one input unit. A fully connected layer from 784 inputs to 256 outputs therefore has fan-in 784 and fan-out 256.

If a layer computes z = W x + b, a useful first approximation is Var(z) ≈ fan_in × Var(W) × Var(x). This assumes the weights and inputs are roughly independent and centred around zero. The approximation is not a law of nature, but it tells us what initialization is trying to control: the scale of the signal before it gets repeatedly transformed.

The activation function matters because it changes that scale. Tanh compresses large values toward negative one or positive one. Sigmoid compresses them toward zero or one, and its derivative becomes very small in either saturated region. ReLU sets every negative input to zero and passes positive inputs through unchanged. The same weight variance cannot be ideal for all three.

The mechanism behind Xavier

Xavier initialization was designed to keep forward activations and backward gradients at comparable scales for approximately linear, centred activations. Its usual weight variance is:

Var(W) = 2 / (fan_in + fan_out)

The corresponding normal distribution uses standard deviation:

sqrt(2 / (fan_in + fan_out))

Its uniform version samples from:

[-sqrt(6 / (fan_in + fan_out)), +sqrt(6 / (fan_in + fan_out))]

Why both fan values? Forward propagation depends on how many inputs contribute to each output, so it depends on fan-in. Backpropagation sends gradients through the transpose-like path, so its scale depends on fan-out. Xavier compromises between those two requirements.

That makes Xavier a sensible default for tanh and linear layers. It is also the conventional choice for sigmoid layers, although sigmoid itself is often a poor choice in a deep network because its saturated regions still produce weak gradients. Xavier prevents an unnecessarily broad initial distribution; it does not repeal the sigmoid derivative.

The mechanism behind He

He initialization is tuned for ReLU. Suppose the preactivation is centred and roughly symmetric. ReLU removes the negative half, so its output is zero for about half the inputs. In terms of the second moment, the average squared size of a value, the ReLU output is approximately half the preactivation second moment.

To compensate, He initialization uses:

Var(W) = 2 / fan_in

The normal version therefore has standard deviation:

sqrt(2 / fan_in)

The uniform version uses bounds:

[-sqrt(6 / fan_in), +sqrt(6 / fan_in)]

The factor of two is the important part. If the preactivation has second moment q, multiplying by weights with variance 2 / fan_in gives a preactivation scale of roughly 2q. ReLU then removes about half of that squared scale, returning it to roughly q.

The same reasoning helps with gradients. A ReLU derivative is zero for a negative preactivation and one for a positive one. Under the same symmetry assumption, about half of the gradient paths remain active. He’s larger weight variance compensates for that loss.

For LeakyReLU or PReLU, the negative side is not completely removed, so the exact factor changes. If the negative slope is a, the common approximation is:

Var(W) = 2 / (fan_in × (1 + a²))

Frameworks such as PyTorch can apply this activation-aware gain when using Kaiming initialization.

A numerical example

Take the first hidden layer of a classifier:

784 inputs → 256 hidden units → ReLU

Assume the input features have a roughly unit scale.

For Xavier normal initialization:

sqrt(2 / (784 + 256)) = 0.0439

For He normal initialization:

sqrt(2 / 784) = 0.0505

The difference in standard deviation looks modest, but the downstream scale differs for a reason. Xavier gives the weights variance 2 / 1040, so the preactivation second moment is approximately:

784 × 2 / 1040 = 1.51

After ReLU, the approximation gives about 0.75. He gives:

784 × 2 / 784 = 2.00

After ReLU, that returns to about 1.00.

For an equal-width ReLU layer, the contrast is clearer. With fan-in equal to fan-out equal to 512, Xavier uses variance 1 / 512. Its preactivation scale is preserved, but ReLU halves the squared scale at every layer. After ten such layers, the simple approximation gives 0.5^10, or about 0.00098, of the starting squared scale. He uses 2 / 512, which compensates for the ReLU gate.

Here is a practical PyTorch pattern for ReLU hidden units followed by a linear logits layer:

import torch.nn as nn

hidden = nn.Linear(784, 256)
nn.init.kaiming_normal_(
    hidden.weight,
    mode="fan_in",
    nonlinearity="relu",
)
nn.init.zeros_(hidden.bias)

logits = nn.Linear(256, 10)
nn.init.xavier_uniform_(logits.weight)
nn.init.zeros_(logits.bias)

The output layer is linear, so it does not need the ReLU compensation. Do not choose He for it merely because the hidden layers use ReLU.

The production rule

Activation after the layer	Good first choice	Important qualification
Linear	Xavier	A reasonable general-purpose choice
Tanh	Xavier	Use an activation-aware gain when appropriate
Sigmoid	Xavier	Usually avoid many stacked sigmoid layers
ReLU	He, usually fan-in mode	Compensates for the inactive negative half
LeakyReLU or PReLU	He with the negative slope	The slope changes the gain
GELU or SiLU	He is a common baseline	The ReLU derivation is only approximate

For convolutional layers, fan-in and fan-out include the receptive field. A 3 by 3 convolution with 64 input channels has fan-in 3 × 3 × 64 = 576, not merely 64.

The default choice for an ordinary feed-forward network is usually fan-in mode because it preserves the forward signal. Fan-out mode instead prioritises backward gradient scale. That can be useful in a specialised architecture, but changing it casually can fix one direction while disturbing the other.

The senior-level nuance

The activation function is the first decision, not the only one. Batch normalization or layer normalization can reduce sensitivity to initial scale, but it does not make initialization irrelevant. Residual connections add paths together, so their variance depends on the branch structure as well as the layer weights. Transformer blocks using GELU, normalization, and residual additions often follow architecture-specific conventions. Blindly replacing those conventions with He because “the network is deep” is not a design principle.

He is also not automatically better for every nonlinearity that is not tanh. GELU and SiLU do not simply throw away exactly half of their inputs. He is often a good starting point because their behaviour is ReLU-like, but validation still matters.

A failure mode you can recognise

Using Xavier throughout a deep, equal-width ReLU network can make later activations nearly zero. The first symptom is usually not an exception. It is a flat training loss, shrinking gradient norms, or layers whose output statistics have an RMS close to zero. Inspect activation RMS, gradient RMS, and the fraction of zero ReLU outputs by layer after the first few batches.

The opposite mistake is using a broad initialization with sigmoid. Large initial preactivations push sigmoid outputs close to zero or one, where the derivative is tiny. The model then shows very slow learning even though the forward pass contains no NaNs. Weight initialization and learning rate can produce similar symptoms, so layer-wise statistics are more informative than guessing from the loss curve alone.

What they’ll ask next

Why not initialise every weight to zero?
All units in a layer would receive the same gradient and remain identical. Random weights break that symmetry. Zero biases are usually fine because the weights already distinguish the units.

When would you use fan-out mode for He initialization?
Use fan-in when preserving forward activation scale is the priority. Fan-out preserves the backward scale instead. It can be useful in architectures where gradient flow is the main concern, but the choice should follow the architecture and measured behaviour rather than a universal slogan.

Does BatchNorm make Xavier versus He irrelevant?
No. It can make the difference smaller by renormalising activations, but the initial signal, gradients, residual branches, and layers before normalization still depend on the initialization. He for ReLU remains the sensible baseline unless the architecture specifies something else.

Say this in the interview

“I choose Xavier for roughly centred linear, tanh, or sigmoid layers, and He for ReLU-family layers: Xavier balances fan-in and fan-out, while He uses the extra variance needed after ReLU removes about half the signal.”