Why does weight initialization matter and how do Xavier and He initialization work?

How to think about it

The short answer is that initialization sets the scale of every signal and gradient before learning has had a chance to correct anything. Xavier, or Glorot, initialization is usually suited to roughly symmetric activations such as tanh; He, or Kaiming, initialization uses a larger variance for ReLU-family activations because ReLU sets negative inputs to zero.

Why initialization matters

Consider one neuron in a layer. It receives fan_in inputs, where fan-in means the number of values entering that neuron. Its preactivation is a weighted sum:

z = w_1 x_1 + w_2 x_2 + ... + w_n x_n

Assume, as a useful approximation, that the inputs and weights are independent, centered around zero, and have finite variance. If the input variance is Var(x) and the weight variance is Var(w), then:

Var(z) ≈ fan_in × Var(w) × Var(x)

That multiplication is the whole problem.

If the product is greater than one in every layer, activations grow as they move forward. A few layers may produce values in the tens, then hundreds. With tanh or sigmoid, those large values push the activation into saturation, where its derivative is close to zero. With an unconstrained output, the values can simply explode.

If the product is less than one, the signal shrinks at every layer. The first layer may look healthy, while the tenth produces values indistinguishable from zero. The gradients flowing back through those layers shrink too, so early weights barely move. This is the vanishing-gradient problem.

Backpropagation has the same issue in reverse. A gradient is multiplied by weight matrices and activation derivatives as it travels toward the input. For a ReLU, the derivative is either zero or one. For sigmoid and saturated tanh, it is often much smaller than one. Initialization therefore affects both the forward signal and the backward learning signal.

A useful target is to keep the scale approximately stable from one layer to the next. It does not need to be exactly constant. Neural networks are not Swiss watches. But a factor of 0.5 repeated across 20 layers is not a small imperfection.

How Xavier initialization works

For a linear layer, preserving the forward variance suggests:

Var(w) ≈ 1 / fan_in

Preserving the backward gradient variance suggests:

Var(w) ≈ 1 / fan_out

Here, fan-out means the number of values produced by the layer. A single variance cannot usually satisfy both conditions when fan-in and fan-out differ, so Xavier chooses a compromise:

Var(w) = 2 / (fan_in + fan_out)

The normal version samples weights with standard deviation:

std = sqrt(2 / (fan_in + fan_out))

The uniform version samples from:

[-sqrt(6 / (fan_in + fan_out)), +sqrt(6 / (fan_in + fan_out))]

The bounds differ because a uniform distribution on [-a, a] has variance a² / 3.

The derivation assumes the activation behaves approximately like a linear function near the values the network initially produces. That makes Xavier a natural starting point for linear layers and tanh, whose derivative is near one around zero. It is also commonly used for sigmoid layers, although sigmoid can still saturate because its derivative is small away from its center.

“Xavier preserves variance” is not a guarantee that every activation will have exactly the same variance. It is a statistical approximation based on assumptions about independence, centering, and the activation distribution.

How He initialization works

ReLU is defined as max(0, z). For an approximately symmetric, zero-centered preactivation, about half the values are negative and become zero.

That means the ReLU output carries roughly half the input’s squared signal. To compensate, He initialization doubles the variance:

Var(w) = 2 / fan_in

The normal version therefore uses:

std = sqrt(2 / fan_in)

The uniform version uses:

[-sqrt(6 / fan_in), +sqrt(6 / fan_in)]

The phrase “ReLU halves the variance” is convenient but slightly imprecise. ReLU creates a positive mean, so the exact variance is not simply half the input variance. The derivation is more accurately tracking the second moment, the average squared activation. Under the symmetric-input approximation:

E[ReLU(z)²] ≈ 0.5 × E[z²]

Choosing Var(w) = 2 / fan_in makes the preactivation second moment about twice as large, and ReLU brings it back to roughly its previous scale.

For a leaky ReLU with negative slope a, the Kaiming family adjusts the variance to approximately:

2 / ((1 + a²) × fan_in)

The actual negative slope must be supplied when using a framework initializer. Using the plain ReLU setting for a very different activation is not automatically correct.

A concrete numerical example

Imagine a multilayer perceptron with four hidden layers. Each layer has 512 units and uses ReLU. Suppose the input activation energy, its average squared value, is 1.

Because the layers have equal width, Xavier gives:

Var(w) = 2 / (512 + 512) = 1 / 512

The preactivation energy is then approximately:

512 × (1 / 512) × 1 = 1

ReLU keeps roughly half of that energy, so the next layer starts with about 0.5. Repeating this gives:

Hidden layer	Approximate activation energy with Xavier
1	1
2	0.5
3	0.25
4	0.125

After ten ReLU layers, the same simplified calculation gives 0.5^10, or about 0.00098. The corresponding root-mean-square activation is only about 0.031. The network has not “learned badly” yet. It started by making the signal tiny.

He initialization gives:

Var(w) = 2 / 512 = 0.00390625

The preactivation energy is approximately:

512 × (2 / 512) × 1 = 2

ReLU halves that back to roughly 1, so the signal remains at a usable scale across layers.

In PyTorch, an explicit ReLU initialization looks like this:

import torch.nn as nn

layer = nn.Linear(512, 512)

nn.init.kaiming_normal_(
    layer.weight,
    mode="fan_in",
    nonlinearity="relu",
)
nn.init.zeros_(layer.bias)

For a tanh layer, a common starting point is:

nn.init.xavier_normal_(layer.weight)
nn.init.zeros_(layer.bias)

Practical choices and their limits

Activation or layer	Usual starting point
Linear or tanh	Xavier
ReLU	He with fan_in
Leaky ReLU	He with the actual negative slope
SELU	LeCun normal under self-normalizing assumptions
Transformer blocks	The architecture’s prescribed initializer

That last row matters. There is no universal “Transformer initialization.” Attention projections may use Xavier, while a particular implementation uses a truncated normal distribution or depth-dependent residual scaling. Follow the architecture and framework documentation rather than applying He to every matrix that happens to be near an attention block.

For convolutional layers, fan-in includes the receptive field. A convolution with 64 input channels and a 3 × 3 kernel has a fan-in of 64 × 3 × 3, adjusted for groups by the framework’s weight layout. For custom layers, verify how fan-in and fan-out are calculated. A transposed weight convention can silently reverse them.

mode="fan_in" is the usual He choice when the goal is to preserve forward activations. fan_out can be useful when preserving backward signal is the stronger concern. This is a design choice, not a spelling error.

Normalization layers and residual connections make networks less sensitive to initialization, but they do not make it irrelevant. Normalization may control activation scale while gradients, residual branches, or the first few optimization steps remain poorly scaled. Initialization also cannot repair unnormalized input features, a learning rate that is 100 times too large, or a mislabeled objective.

If using transfer learning, do not reinitialize the pretrained layers. Initialize only newly added layers unless there is a deliberate reason to discard the learned representation.

Common traps and failure symptoms

Common trap: setting every hidden weight to zero, or to the same constant. Every neuron then computes the same function and receives the same gradient, so the neurons remain identical. The network effectively wastes its width. Random values are the usual way to break this symmetry, but randomness itself is not sacred; distinct orthogonal weights can also break it.

Zeroing the output layer can sometimes be less harmful because output neurons may receive different gradients from different target coordinates. That exception does not make zero initialization safe for hidden layers.

A practical failure often appears first in activation statistics, not in accuracy. With Xavier initialization accidentally used throughout a deep ReLU network, early layers may have normal-looking values while later layers show rapidly falling standard deviations, a high fraction of zeros, and gradient norms close to zero. Training loss then plateaus.

The opposite symptom is a loss that becomes nan or inf on the first few batches. Activation standard deviations and gradient norms grow from layer to layer. An overly large initialization, an incorrect fan calculation, or a wrong tensor orientation is a likely suspect.

A useful smoke test is to run one batch before training and log, for each layer:

• activation mean and standard deviation;
• the fraction of zero outputs for ReLU;
• gradient norms after one backward pass;
• whether any value is nan or inf.

These numbers need not match perfectly. They should not change by several orders of magnitude from one layer to the next.

What they’ll ask next

Why not use He initialization for every network?

Because its factor of two is designed for the gating behavior of ReLU-like activations. With tanh or sigmoid, that larger scale can push preactivations into saturation, where derivatives are small. Xavier is a better default for roughly symmetric activations, though the final choice should be validated with activation statistics.

Why does Xavier use fan-out at all?

Forward activations depend mainly on fan-in. Backward gradients depend on how many terms contribute to each gradient, which involves fan-out. Xavier balances those two goals with 2 / (fan_in + fan_out). If only forward stability matters, 1 / fan_in is the simpler target.

Does PyTorch’s default nn.Linear initialization automatically choose He for a following ReLU?

No. PyTorch’s nn.Linear default uses a Kaiming-uniform helper configuration equivalent to a weight bound of 1 / sqrt(fan_in). That is not the usual ReLU He variance of 2 / fan_in. If the activation matters, initialize the layer explicitly or use a model implementation whose initialization is designed for that architecture.

Say this in the interview

“Initialization controls whether activations and gradients stay at a learnable scale; Xavier balances fan-in and fan-out for roughly symmetric activations, while He uses 2 / fan_in because ReLU removes about half the signal.”