Weight initialization

A fresh model has random weights, a random seed, and absolutely no idea what a cat is. That is normal.

What is not normal is a fresh model whose first forward pass produces activations of size 10^18, or whose hidden units all produce exactly the same number. The optimizer has not had a chance to help. The network has already lost.

In the training-loop demo we started one neuron at w = 0, b = 0 and it learned fine. In a plain feed-forward ReLU network with all hidden weights and biases set to zero, hidden gradients are typically zero, so the network cannot learn useful features. Zeroing only the output layer, using nonzero biases, or using residual paths changes the result. Initialization is the least glamorous choice in deep learning and one of the most decisive: get the scale wrong and your gradients are dead before the first step.

The problem: variance compounds with depth

Imagine a 256-wide network with 15 fully connected ReLU layers. The input to the first layer has a standard deviation of 1. A standard deviation measures the typical distance from the mean; it describes the scale of a signal.

Each neuron computes a weighted sum:

z = w₁x₁ + w₂x₂ + ... + wₙxₙ

If the inputs are roughly independent, centered around zero, and have variance q, then:

Var(z) ≈ fan_in × Var(w) × q

fan_in is the number of values entering one neuron. The variance contributions from independent terms add, and larger weights contribute more because variance scales with the square of weight size.

Repeat this operation through 15 layers and the effect compounds. A factor above 1 enlarges the signal at every layer; a factor below 1 repeatedly shrinks it. The same multiplication happens backward: gradients pass through each weight matrix and activation derivative, so they can explode or vanish.

With weights drawn from N(0, 1), the weight variance is 1. In our 256-wide example, the first pre-activation has a standard deviation of roughly:

sqrt(256 × 1 × 1) = 16

That is already large. ReLU keeps positive values and replaces negative values with zero, so the next matrix receives a large signal too. A 15-layer float64 run can reach scales around 10^15; with more depth or lower precision, values may overflow to inf, followed by NaN.

Weights that are too small fail in the other direction. If every layer reduces the scale by 0.8, then after 15 layers it is 0.8^15, or about 0.035 of the initial scale. At 100 layers it is about 2.0 × 10^-10.

For ReLU, ordinary variance is not quite the quantity the classic derivation preserves because ReLU outputs are no longer centered at zero. The useful calculation tracks the second moment, the expected squared activation. If a zero-mean Gaussian enters ReLU, roughly half its values survive:

E[ReLU(z)^2] ≈ 1/2 × E[z^2]

That factor of one-half is why He initialization contains a 2.

The first trap: all-zero weights

Suppose every hidden neuron starts with all its weights set to zero. Every neuron receives the same input, computes the same output, and—under the usual symmetric downstream setup—receives the same gradient. After the update, every neuron still has the same weights. A layer with 1,000 neurons has the learning capacity of one neuron wearing a large hat.

Zeroing only an output layer can sometimes be harmless: its weights can receive a useful gradient while the hidden representation remains fixed for the first update. Zeroing a hidden layer, or every weight in the network, is the classic failure. Randomize hidden weights; zero biases unless the architecture requires otherwise.

Random does not mean enormous. N(0, 1) breaks symmetry, but its scale is wrong for a wide layer. Initialization needs both asymmetry and an appropriate variance.

Xavier and He: choosing the scale

The goal is for the next layer to see a signal with approximately the same scale as the previous one. The activation function determines the appropriate scale.

He initialization for ReLU

For a ReLU layer, the second-moment calculation is:

q_next ≈ fan_in × Var(w) × q / 2

To make q_next approximately equal to q, choose:

Var(w) = 2 / fan_in

This is He initialization, also called Kaiming initialization. Its normal distribution has standard deviation:

sqrt(2 / fan_in)

For a 256-input layer:

• He variance is 2 / 256 = 0.0078125.
• He standard deviation is about 0.0884.
• A weight with standard deviation 1 is more than 11 times larger.

The factor of 2 compensates for ReLU discarding roughly half the squared signal. For leaky ReLU with negative slope a, the approximate variance is:

2 / ((1 + a^2) × fan_in)

Xavier for tanh and sigmoid

For an activation that is approximately linear around zero:

q_next ≈ fan_in × Var(w) × q

Xavier, or Glorot, uses a fan-in/fan-out compromise:

Var(w) = 2 / (fan_in + fan_out)

fan_out is the number of neurons receiving the layer’s outputs. Forward propagation favors one scale, while backward propagation favors another, so Glorot balances the two.

Use He for ReLU and close relatives, and Xavier for tanh. Xavier is the classic choice for sigmoid, but deep sigmoid networks can still saturate because its derivatives become small away from zero. GELU, SwiGLU, and other modern activations need their architecture’s established recipe rather than a blindly applied ReLU rule.

In PyTorch, explicit He initialization for a ReLU linear layer looks like this:

import torch.nn as nn

layer = nn.Linear(512, 512)
nn.init.kaiming_normal_(layer.weight, mode="fan_in", nonlinearity="relu")
nn.init.zeros_(layer.bias)

fan_in preserves forward activation scale. Check the actual input dimension when using a custom layer: a correct formula applied to the wrong axis is still wrong.

Residual networks change the arithmetic

A plain stack replaces one representation with the next:

x_next = F(x)

A residual block adds a branch:

x_next = x + F(x)

The identity term provides a direct forward and backward path, while the branch learns a correction. But independent branch variances add:

Var(x_next) ≈ Var(x) + Var(F(x))

If each branch is comparable in variance to the current stream, the recurrence is roughly V_{l+1} ≈ 2V_l, which grows exponentially. If each branch has a fixed variance σ_F^2, then:

V_{l+1} = V_l + σ_F^2

After L additions, that is approximately V_0 + Lσ_F^2: linear variance growth, or square-root growth in standard deviation. With M independent branches, scaling each output by α gives accumulated variance Mα^2σ_F^2; α ≈ 1 / sqrt(M) keeps it bounded under this assumption.

Residual architectures therefore often make each writing branch smaller, using a factor proportional to 1 / sqrt(L) or zero-initializing its final projection. The exact constant depends on the number of branches, normalization placement, and whether the model is pre-norm or post-norm. Scale the residual contribution or final projection rather than blindly shrinking every internal weight.

Normalization parameters commonly start with scale one and shift zero. This is why initialization and normalization layers are often designed together.

What fails first in practice

Initialization bugs often appear on the first batch:

• NaN or inf immediately: inspect activation and gradient ranges after the first forward and backward pass. An output standard deviation of 10^8 from input scale 1 suggests an incorrect weight scale, missing normalization, or the wrong fan dimension. Mixed precision makes overflow easier.
• Finite activations but zero early gradients: saturated tanh outputs or mostly inactive ReLUs indicate that signals or derivatives have shrunk. Match the initializer to the activation, then check learning rate and input scale.
• Identical hidden features: inspect weights after one or two updates. Rows that remain identical indicate symmetry—often a zeroing reset or checkpoint load. Use random asymmetric hidden weights, not randomly huge ones.
• Reasonable forward scale but divergent training: inspect gradient norms by depth too. Rectangular or custom layers may need fan-out or a Glorot compromise instead of the usual fan-in rule.
• A pretrained model performs terribly after a new run: do not reinitialize the learned backbone. Initialize only newly added parameters. This is the difference between initialization and fine-tuning.

The honest limits

Xavier and He are moment-matching methods. They control average scale, not every singular value or information-carrying direction. Inputs become correlated during training, and attention, gating, normalization, convolution, weight sharing, and residual additions change the assumptions.

Normalization and residual paths make initialization less dominant in some modern architectures, but framework defaults are not magic. A custom layer, unusual activation, or very deep residual stack can invalidate them.

The practical approach is:

1. Use the framework default for standard layers unless you have a reason to override it.
2. Match the initializer to the activation and actual fan dimensions.
3. Measure activation and gradient scales on the first batch.
4. Follow architecture-specific residual recipes.
5. Never reinitialize learned parameters just because a new run began.

In one breath

• Per-layer scale errors compound through matrix multiplications, causing exploding or vanishing activations and gradients.
• For ReLU, He uses Var(w) = 2 / fan_in to replace the squared signal ReLU removes.
• Xavier uses Var(w) = 2 / (fan_in + fan_out) and is the classic tanh and sigmoid choice.
• Randomness breaks symmetry; scale controls propagation. Zero biases are usually fine, but zero hidden weights are not.
• These methods control starting scale under simplifying assumptions. They do not replace normalization, sensible learning rates, or residual-specific recipes.

Quick check

Next

Good initialization gives the gradients a fighting start. When a run still stalls or blows up, measure what each layer is doing rather than staring only at the loss: debugging a training run covers that diagnostic loop.