Activation functions

A neural network with no activation functions is just matrix multiplication — and stacked matrix multiplications collapse to a single matrix multiplication. So without nonlinearities, a 100-layer network has the same expressive power as a 1-layer network. Activations break that linearity and are what make depth useful.

The collapse, demonstrated

import numpy as np

rng = np.random.default_rng(0)
W1 = rng.standard_normal((4, 8))
W2 = rng.standard_normal((8, 8))
W3 = rng.standard_normal((8, 2))
x = rng.standard_normal((1, 4))

# Stacked linear: x @ W1 @ W2 @ W3
deep_linear = x @ W1 @ W2 @ W3

# Equivalent single matrix
W_combined = W1 @ W2 @ W3
single_linear = x @ W_combined

print("3-layer linear:", deep_linear.round(4))
print("1-layer linear:", single_linear.round(4))
print("identical?    ", np.allclose(deep_linear, single_linear))

3-layer linear: [[0.678  0.3721]]
1-layer linear: [[0.678  0.3721]]
identical?     True

Three layers, one effective transformation. Add a nonlinearity between them and the network can learn curved decision boundaries, gating behaviors, and the rich function classes deep learning is famous for.

The hidden-layer activations

These go between layers. They’re the ones you’ll write nn.ReLU() or nn.GELU() for.

import numpy as np

x = np.linspace(-4, 4, 9)

def relu(x):        return np.maximum(0, x)
def leaky_relu(x):  return np.where(x > 0, x, 0.01 * x)
def gelu(x):
    # Approximate (tanh-based) GELU — what most implementations use
    return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))
def silu(x):        return x / (1 + np.exp(-x))   # also called Swish

print(f"{'x':>5} {'ReLU':>7} {'LeakyReLU':>10} {'GELU':>8} {'SiLU':>8}")
for xi in x:
    print(f"{xi:5.1f} {relu(xi):7.3f} {leaky_relu(xi):10.3f} {gelu(xi):8.3f} {silu(xi):8.3f}")

    x    ReLU  LeakyReLU     GELU     SiLU
 -4.0   0.000     -0.040   -0.000   -0.072
 -3.0   0.000     -0.030   -0.004   -0.142
 -2.0   0.000     -0.020   -0.045   -0.238
 -1.0   0.000     -0.010   -0.159   -0.269
  0.0   0.000      0.000    0.000    0.000
  1.0   1.000      1.000    0.841    0.731
  2.0   2.000      2.000    1.955    1.762
  3.0   3.000      3.000    2.996    2.858
  4.0   4.000      4.000    4.000    3.928

A quick tour of what each one’s for:

The table tells you what each one does; the real difference is in the gradient. Each activation is only as useful as the gradient it lets flow backward during training — that derivative f'(x) is the signal backprop multiplies through, and where it goes to zero, learning stalls.

ReLU’s key advantage: its derivative is exactly 1 for all positive inputs, so gradients pass through unchanged on the positive side. Sigmoid and tanh, by contrast, squash their entire input range into a small output range — for large |x| the derivative is nearly zero, shrinking every upstream gradient. Stack 20 layers of sigmoid and that gradient signal becomes too faint to update early layers (the vanishing gradient problem). ReLU avoids this for positive activations entirely.

One hazard: a ReLU neuron whose input is always negative will always output 0 and its gradient is always 0 — it can never recover. This is a dead neuron. Leaky ReLU fixes it with a small negative slope; GELU/SiLU fix it smoothly.

The output activations

These go at the end of the network, shaping the output to match the task.

Sigmoid — 1 / (1 + exp(-x)) — squashes to (0, 1). Use for binary classification outputs (probability of class 1), or for any “this is a probability of one event” output. Used to be common in hidden layers — it isn’t anymore (gradients vanish at the tails).

Tanh — (e^x - e^-x) / (e^x + e^-x) — squashes to (-1, 1). Like sigmoid but centered. Mostly historical now except in some RNN variants and specific output scaling.

Softmax — exp(x_i) / Σ exp(x_j) — turns a vector of logits into a probability distribution that sums to 1. Use for multi-class classification: the network outputs one logit per class, softmax turns them into probabilities.

import numpy as np

def sigmoid(x):    return 1 / (1 + np.exp(-x))
def softmax(x):
    # Numerically stable: subtract max before exp
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

# Binary classification: one logit -> probability
binary_logit = np.array([2.0])
print("sigmoid(2.0) =", sigmoid(binary_logit).round(3))

# Multi-class (3 classes): a vector of logits -> probs that sum to 1
multi_logits = np.array([1.0, 3.0, 0.2])
probs = softmax(multi_logits)
print("softmax probs:", probs.round(3), "sum:", probs.sum())

sigmoid(2.0) = [0.881]
softmax probs: [0.113 0.836 0.051] sum: 0.9999999999999999

Picking one (the short version)

• Hidden layers: SiLU or GELU. ReLU if you want cheap and don’t care.
• Binary output: sigmoid.
• Multi-class output: softmax (often fused with cross-entropy loss as nn.CrossEntropyLoss, which expects raw logits — don’t apply softmax yourself!).
• Regression output: nothing. Output the raw values.

In one breath

• Without nonlinear activations a deep network collapses to a single matrix multiply — depth only helps once you break the linearity.
• Hidden layers: ReLU (cheap, sparse, derivative 1 on the positive side) gave way to GELU/SiLU (smooth, slightly negative near 0) in transformers; SiLU inside SwiGLU dominates modern LLMs.
• An activation is only as good as the gradient it passes: ReLU avoids vanishing on the positive side but can leave dead neurons; smooth activations keep the gradient alive everywhere.
• The output layer matches the task: sigmoid for binary, softmax for multi-class single-label, nothing for regression.
• With nn.CrossEntropyLoss output raw logits — it applies log-softmax internally; applying softmax yourself is a bug.

Quick check