Activation functions
ReLU, GELU, SiLU, sigmoid, and softmax: what each computes, why nonlinearities make depth useful, and how to match an activation to the task and loss.
Written and reviewed by Shreyash Prashu Last reviewed
What you'll learn
- Why stacked linear layers collapse and nonlinearities prevent that
- How activation derivatives control gradient flow during training
- When ReLU, GELU, SiLU, sigmoid, tanh, and gated activations make sense
- How to match output activations with binary, multilabel, multiclass, and regression losses
- How to diagnose dead neurons, saturation, NaNs, and logits-loss mistakes
Before you start
Your warehouse camera has learned to classify fruit. It sees a photograph, pushes numbers through twenty layers, and reports three scores: ripe, under-ripe, and rotten.
Now remove the activation functions between those layers.
The model still has twenty matrix multiplications and millions of parameters.
But mathematically, all those linear transformations merge into one. The network can:
- rotate the input,
- stretch the input,
- shift the input.
But it cannot bend a decision boundary around a cluster of rotten apples.
Activation functions are the bends. Their nonlinearity lets one layer change what the next layer means. That is what makes depth useful.
It's the gradient that makes or breaks an activation
Each activation plots its value f(x) and its derivative f′(x) on the same axes. Drag the input x marker into the tails of Sigmoid/Tanh and watch the gradient collapse toward 0; on ReLU, slide it below 0 into the dead zone. That gradient story is why ReLU, GELU, and SiLU took over deep nets.
Squashes to (0, 1). Both tails saturate, so f′ collapses toward 0 — gradients vanish and deep stacks barely learn. Lives on now as a binary-output activation, not a hidden one.
Why linear layers collapse
A neuron first computes an affine transformation—a weighted sum plus a bias: z = xW + b. An activation transforms it into a = f(z).
Without f, a second layer applies another affine transformation, and the composition is still affine:
(xW1 + b1)W2 + b2 = x(W1W2) + (b1W2 + b2).
The biases change the offset, not the central fact: a stack of affine layers is equivalent to one affine layer. This is why people often say that the matrix multiplications “collapse.”
Here is a small network. The first two layers have two values each; the final layer produces one value.
import numpy as np
x = np.array([[1.0, -2.0]])
W1 = np.array([[2.0, 1.0],
[-1.0, 3.0]])
W2 = np.array([[1.0, -2.0],
[4.0, 0.0]])
W3 = np.array([[2.0],
[1.0]])
stacked = x @ W1 @ W2 @ W3
combined = x @ (W1 @ W2) @ W3
h1 = np.maximum(0, x @ W1)
h2 = np.maximum(0, h1 @ W2)
with_relu = h2 @ W3
print("stacked:", stacked)
print("combined:", combined)
print("with ReLU:", with_relu)stacked: [[-40.]]
combined: [[-40.]]
with ReLU: [[8.]]The first two results are identical. ReLU interrupts the composition by turning negative values into zero before the next layer sees them. The later layer therefore receives information that one global affine transformation could not represent.
This lets a network build piecewise-linear regions, thresholds, and gates. A ReLU network is made of straight pieces, but enough pieces can approximate complicated shapes. Without nonlinearities, it is a shallow affine model wearing a deep-network costume.
The gradient is the mechanism
Forward computation explains expressive power. Backpropagation explains why activation shape affects learning.
The chain rule multiplies derivatives as the error signal travels backward. If a = f(z), the activation contributes f'(z). A derivative near zero shrinks the gradient passed to earlier layers; one near one passes it along at roughly its current size.
For sigmoid,
sigmoid(z) = 1 / (1 + exp(-z))
and
sigmoid'(z) = sigmoid(z) × (1 - sigmoid(z)).
Its largest derivative is 0.25, at zero. At z = 8, sigmoid is about 0.9997 and its derivative about 0.0003; at -8, the derivative is similarly tiny.
The unit is saturated, so changing its input barely changes its output. Across twenty layers, those small factors can produce vanishing gradients.
ReLU instead computes:
ReLU(z) = max(0, z).
For positive inputs its derivative is 1; for negative inputs it is 0. It avoids saturation on the positive side, but a neuron that stays negative always outputs zero and receives no gradient. This is a dead neuron.
Leaky ReLU gives the negative side a small slope, commonly 0.01, so its gradient is small rather than zero. GELU and SiLU use smooth curves with small negative outputs.
Smoothness means a gradual transition around zero, not a large negative-side derivative.
The derivative is not the only consideration. Activations also change:
- value distributions,
- sparsity,
- numerical cost,
- architectural behavior.
Initialization, normalization, learning rate, and data scale can matter more than swapping between sensible modern choices. Gradient problems covers the broader causes and remedies.
Hidden-layer activations
Hidden activations sit between learned layers. Common choices include:
| Activation | Definition | Common use | Main trade-off |
|---|---|---|---|
| ReLU | max(0, z) | CNNs, MLPs, lightweight models | Cheap and clean positive gradients; negative inputs receive no gradient |
| Leaky ReLU | z or αz, with small α | Models where dead ReLUs are troublesome | Preserves a negative gradient, but adds a slope choice |
| GELU | z × Φ(z) | Transformer encoders and older Transformer families | Smooth and effective; costs more than ReLU |
| SiLU / Swish | z × sigmoid(z) | Transformer blocks, CNNs, MLPs | Smooth and often strong; creates fewer exact zeros |
| tanh | (exp(z) - exp(-z)) / (exp(z) + exp(-z)) | Some recurrent models and bounded signals | Zero-centered, but saturates at both extremes |
GELU may use an exact or tanh-based approximate formula. Check the framework setting when matching a published checkpoint.
SiLU is also called Swish.
ReLU’s zeros can create sparse activations, but zeros in a dense matrix multiplication do not automatically make modern hardware skip the multiply. Dense fused kernels may still be faster.
Smooth activations reduce the risk of permanently silent units, but they are not universally better: a small CPU MLP may value ReLU’s cost more than a modest quality improvement. Measure the model you actually have.
What large Transformer blocks do
Many modern language models use a gated feed-forward block rather than a plain hidden activation:
h = (f(xW1) elementwise-multiplied-by xWg)W2.
The gate lets some channels through and suppresses others for a particular token. With SiLU, this is commonly called SwiGLU; related GELU designs are called GEGLU.
The extra projection and gate add compute, so designers usually adjust hidden width to keep the total budget reasonable.
Llama, Mistral, Qwen, and many other recent Transformer families use gated variants. GELU remains common in encoder-style and older Transformer checkpoints. ReLU remains a sensible baseline for CNNs, compact MLPs, and cost-sensitive systems.
Output activations are a contract with the task
The final layer must express the kind of answer the task permits.
Softmax for mutually exclusive classes
For three mutually exclusive fruit classes, output three raw scores called logits. During training, use the logits directly; softmax is for probabilities at inference:
softmax(zi) = exp(zi) / sum(exp(zj)).
For logits [1.0, 3.0, 0.2], subtract the largest value before exponentiating:
- shifted logits:
[-2.0, 0.0, -2.8] - exponentials: approximately
[0.1353, 1.0000, 0.0608] - total: approximately
1.1961 - probabilities: approximately
[0.113, 0.836, 0.051]
Subtracting the maximum changes no probability and avoids overflow. The 0.836 is not proof of calibrated confidence.
With PyTorch, nn.CrossEntropyLoss expects raw logits and a target class index. It applies a stable log-softmax internally.
For the example above, if class two is correct, the loss is approximately -log(0.836) = 0.179.
Applying softmax before CrossEntropyLoss is a common bug: the loss expects logits, not probabilities. The code may still run, making the mistake easy to miss.
Sigmoid for one event or independent labels
For a binary task, one raw logit is enough. Sigmoid maps it to (0, 1):
sigmoid(z) = 1 / (1 + exp(-z)).
A logit of 2 becomes approximately 0.881; zero becomes 0.5. Train with nn.BCEWithLogitsLoss, which combines sigmoid and binary cross-entropy stably. Do not apply sigmoid first.
Sigmoid is also correct for multilabel classification, where several labels can be true at once. Give each label an independent logit. Their probabilities do not need to sum to one.
- Multiclass: exactly one class is correct. Use one logit per class with
CrossEntropyLoss. - Multilabel: zero, one, or several labels can be correct. Use one logit per label with
BCEWithLogitsLoss. - Binary: the one-label special case, usually represented by one logit.
You can represent binary classification with two logits and cross-entropy, but do not mix a two-logit head with a one-logit loss.
Regression usually needs no activation
For tomorrow’s temperature, the final output should normally be a raw number. Sigmoid and tanh impose artificial ranges unless you deliberately scale the target and reverse that scaling.
Use a linear output with mean squared error or Huber loss for ordinary unbounded regression. A bounded target can justify a bounded output when the bound comes from the problem. Loss functions explains the interaction with the objective.
Failure modes you can see in a real run
Activation statistics are all zero
A few zero ReLU outputs are normal. Entire channels that remain zero across batches indicate dead neurons. Inspect preactivations and investigate:
- an excessive learning rate,
- poor initialization,
- input scale,
- bias shift.
Leaky ReLU or SiLU may reduce the consequence, but fixing the cause is better.
Hidden sigmoid or tanh saturates
If sigmoid units cluster near zero or one, or tanh units near either extreme, their derivatives are tiny and earlier layers learn slowly.
For ordinary hidden layers, prefer:
- ReLU,
- GELU,
- SiLU.
Also check:
- initialization,
- normalization,
- input scale.
See Normalization layers and weight initialization.
The loss is NaN or infinity
Hand-written exponentials are a prime suspect, especially in mixed precision. Use:
- stable fused losses,
- subtract the maximum for softmax,
BCEWithLogitsLossinstead of sigmoid followed by manual binary cross-entropy.
If the issue persists, inspect:
- the learning rate,
- inputs,
- gradient norms.
Probabilities or regression ranges look wrong
Applying softmax before CrossEntropyLoss, sigmoid before BCEWithLogitsLoss, or independent sigmoid thresholds to a multiclass head produces incorrect semantics. For regression, inspect the final activation if predictions are clipped to a narrow range. Remove it or scale the target deliberately.
In one breath
- Stacked affine layers collapse into one affine transformation; nonlinearities let depth represent bends and gates.
- Derivatives control gradient flow: sigmoid and tanh can saturate, while ReLU can create dead neurons.
- ReLU is a cheap baseline; GELU and SiLU are smooth alternatives, and Transformers often use gated variants.
- Hidden activations are design choices. Output activations follow label semantics.
- Use raw logits with
CrossEntropyLossfor multiclass,BCEWithLogitsLossfor binary or multilabel tasks, and usually no activation for regression.
Quick check
Quick check
nn.CrossEntropyLoss. What should the final layer output?Practice this in an interview
All questionsA ReLU neuron dies when its pre-activation is negative for every relevant training example, so its zero negative-side derivative removes the gradient needed to update its incoming weights and bias. Use Kaiming initialization, a stable learning rate, activation monitoring, and, when needed, leaky ReLU, PReLU, ELU, or GELU to reduce the risk.
GELU, or Gaussian Error Linear Unit, multiplies an input by the standard normal CDF at that input, creating a smooth, input-dependent gate. It often works better than ReLU in Transformer feed-forward layers because it preserves useful gradients around zero, but the advantage is empirical rather than universal and comes with extra compute.
Sigmoid squashes to (0,1) and saturates at extremes, causing vanishing gradients. Tanh is zero-centered but still saturates. ReLU avoids saturation for positive inputs and trains fast but can produce dead neurons. Leaky ReLU fixes dying neurons. GELU is smooth and probabilistic, now the default in most transformer architectures.
The theorem proves that a single-hidden-layer network with enough neurons and a non-linear activation can approximate any continuous function on a compact domain to arbitrary precision. It guarantees existence, not learnability — it says nothing about how many neurons are needed, whether gradient descent will find the solution, or how the network will generalize.
Weight initialization controls the scale of activations and gradients before training begins. Xavier uses variance 2 / (fan-in + fan-out) for roughly symmetric activations, while He uses 2 / fan-in to compensate for ReLU setting negative inputs to zero.
A single artificial neuron computes an affine score by taking a weighted sum of its inputs and adding a bias, then applies an activation function. A classic perceptron uses a hard threshold, while modern neurons commonly use sigmoid or ReLU; the bias shifts the decision boundary and the activation determines the output shape.
A neural network can be built without activations, but stacked affine layers collapse into one affine transformation, so depth adds no functional expressive power. Nonlinear activations let the network represent curved or piecewise decision boundaries and feature interactions, while the right choice depends on the layer and gradient behavior.