Why do neural networks need activation functions at all?

How to think about it

Neural networks do not mathematically require activation functions; a stack of linear layers can run and even be trained. But without a nonlinear activation between layers, the whole stack collapses into one affine transformation, so depth adds no expressive power.

Why

An nn.Linear layer computes an affine transformation, meaning a matrix multiplication followed by a bias addition:

h = W x + b

Here, x is the input, W contains learned weights, and b shifts the result. People often call this a linear layer, although the bias technically makes the transformation affine rather than strictly linear.

Now stack two such layers:

h1 = W1 x + b1
y  = W2 h1 + b2

Substitute the first equation into the second:

y = W2 (W1 x + b1) + b2
  = W2 W1 x + W2 b1 + b2

The two layers are equivalent to:

y = W_eff x + b_eff

That is still one affine transformation. The same algebra works for 100 affine layers. You may have more parameters and a more awkward optimization problem, but the input-output function is no richer than that of a single affine layer.

This is the point an interviewer is usually probing. A candidate who says only “activations add nonlinearity” knows the slogan. A stronger answer explains that composition of affine maps remains affine.

For classification, an affine model can create only a flat decision boundary: a line in two dimensions, a plane in three dimensions, or a hyperplane in higher dimensions. A final sigmoid can turn the score into a probability, but it does not change that boundary. A one-layer network with a sigmoid output is logistic regression. A one-layer network with softmax outputs is a multiclass linear classifier.

The hidden activation is what changes the shape of the function.

A concrete example: XOR

Consider the XOR problem. The label is 1 when exactly one of two binary inputs is 1:

x1	x2	label
0	0	0
0	1	1
1	0	1
1	1	0

The positive examples sit on opposite corners. No single straight boundary can separate them from the two negative examples.

Suppose an affine classifier uses the score z = w1 x1 + w2 x2 + b, and predicts 1 when the score is positive. To classify the four points correctly, it would need all of these conditions:

b < 0
w2 + b > 0
w1 + b > 0
w1 + w2 + b < 0

The first and last conditions say the two negative points must have negative scores. The middle two say the positive points must have positive scores. Those requirements contradict one another. An affine model cannot solve XOR.

Now add the ReLU activation, defined as ReLU(z) = max(0, z). Use two hidden units:

h1 = ReLU(x1 + x2)
h2 = ReLU(x1 + x2 - 1)
score = h1 - 2 h2

Evaluate it on the four inputs:

input	h1	h2	score
(0, 0)	0	0	0
(0, 1)	1	0	1
(1, 0)	1	0	1
(1, 1)	2	1	0

A threshold of 0.5 produces exactly the XOR labels.

The first hidden unit responds to whether the inputs add up to at least 1. The second responds to whether they add up to at least 2. The output combines those two responses so that the (1, 1) case is cancelled out. The activation functions create separate regions of behavior; the final linear layer combines those regions.

That is what “feature interaction” means here. The model is not merely assigning one independent weight to x1 and another to x2. It is treating the same features differently in different parts of the input space.

What changes during training

An activation is usually applied element by element:

h = f(W x + b)

The function f does two jobs.

First, it expands the functions the network can represent. ReLU networks are piecewise linear: each unit is flat for negative inputs and has slope 1 for positive inputs. A large network can combine many such pieces into a complicated overall shape. The individual pieces are simple; their arrangement is not.

Second, the activation changes how gradients flow through the network. Backpropagation uses the chain rule, so the derivative of each activation participates in the gradient.

For ReLU, the derivative is 0 on the negative side and 1 on the positive side. A unit whose preactivation stays negative receives no gradient through that path. This gives ReLU a useful advantage over sigmoid in many hidden layers: positive activations do not continuously shrink the gradient.

Sigmoid behaves differently. Its derivative is:

sigma'(z) = sigma(z) (1 - sigma(z))

The largest possible derivative is 0.25, at z = 0. At z = 8, the derivative is approximately 0.000335. Through many layers, repeatedly multiplying by small derivatives can make gradients extremely small. This is called vanishing gradients.

That does not mean a network without activations cannot learn. A deep linear network can have nonzero gradients and can learn a useful matrix factorization. It simply cannot learn a function outside the affine family. Activations are needed for expressive power, not because gradient descent would otherwise be impossible.

A rough comparison looks like this:

Activation	Common role	Main risk
ReLU	Hidden layers in many MLPs and CNNs	Units can become permanently inactive
GELU	Hidden layers in many Transformer-style models	More computation and no universal guarantee of improvement
Sigmoid	Binary output probabilities and gates	Saturates in deep hidden stacks
Tanh	Bounded, zero-centered hidden or state values	Also saturates
Softmax	Multiclass output probabilities	Usually not used as a generic hidden activation

No activation is “best” in isolation. The architecture, initialization, normalization, task, and loss all matter.

The production pattern

A typical feed-forward classifier places nonlinearities between learned affine transformations:

model = nn.Sequential(
    nn.Linear(2, 8),
    nn.ReLU(),
    nn.Linear(8, 1),
)

The first layer maps the two input features into eight learned intermediate features. ReLU lets each of those features behave differently in different regions. The final layer combines them into one output logit, which is an unnormalized score.

The output layer is task-specific:

• For regression, the final layer is often linear because predictions may need to take any real value.
• For binary classification, the model often returns a logit and applies sigmoid when a probability is needed.
• For multiclass classification, the model often returns one logit per class and applies softmax when probabilities are needed.
• For multi-label classification, each label usually gets its own sigmoid because several labels can be true at once.

A practical detail matters here: many loss functions expect logits rather than probabilities. For example, PyTorch’s BCEWithLogitsLoss combines the sigmoid and binary cross-entropy calculation in a numerically stable way. Applying sigmoid in the model and then passing the result to that loss can produce the wrong setup.

The final layer therefore does not automatically need the same activation as the hidden layers. Adding ReLU to a regression output would prevent negative predictions. Adding sigmoid to every hidden layer would invite saturation. “Put an activation after every layer” is not a rule.

The senior-level nuance

The Universal Approximation Theorem says that a sufficiently wide feed-forward network with a suitable nonlinearity can approximate any continuous function on a compact domain to arbitrary accuracy. That is an existence result, not a promise that training will find the solution, that the network will be small, or that it will extrapolate sensibly outside the training range.

Also, more expressive is not always better. If the relationship is genuinely linear, linear regression or logistic regression may be preferable. They use fewer moving parts, are easier to inspect, often need less data, and can extrapolate more predictably. A neural network with unnecessary nonlinear capacity can fit noise without adding useful signal.

Deep linear networks still have legitimate uses. Their factorized parameterization can impose structure, and optimization may behave differently from directly learning one matrix. But those are optimization or parameterization effects. They do not make the represented function nonlinear.

A ReLU network is another useful subtlety. ReLU is not smooth at zero, yet that is usually acceptable because a single point of nondifferentiability does not prevent gradient-based training. Frameworks assign a subgradient there. If smooth derivatives are important for a particular application, GELU, tanh, or softplus may be a better fit, but smoothness alone does not guarantee better accuracy.

A common failure mode

The first symptom of a missing activation is often a “deep” model whose training curve looks much like a linear baseline and that cannot fit a simple nonlinear pattern such as XOR.

This model has no hidden nonlinearity:

model = nn.Sequential(
    nn.Linear(20, 128),
    nn.Linear(128, 64),
    nn.Linear(64, 1),
)

Despite its width and apparent depth, it represents one affine map from 20 inputs to 1 output. Insert an activation between the layers:

model = nn.Sequential(
    nn.Linear(20, 128),
    nn.ReLU(),
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Linear(64, 1),
)

A different failure appears with ReLU itself. If a unit’s preactivation is negative for every training example, its output is always zero and its gradient is often zero. The symptom is an activation histogram full of zeros and a unit that never changes. Excessive learning rates, poor initialization, or badly scaled inputs can cause this. Lowering the learning rate, fixing input scaling, reinitializing the layer, or using a leaky activation can help.

What they’ll ask next

Can a deep linear network still be trained?
Yes. Its gradients need not vanish, and it may learn a useful factorization of one matrix. But its final input-output function is still affine, so depth does not give it nonlinear decision boundaries.

Why not use sigmoid everywhere if it is nonlinear?
Because sigmoid saturates. For large positive or negative inputs, its derivative is close to zero, so gradients shrink as they pass through layers. It remains useful for binary probabilities and gates, but ReLU-like or GELU-like functions are often easier to optimize in hidden layers.

Should the final layer also have an activation?
It depends on the target. Regression commonly uses no output activation, binary classification commonly produces a logit with sigmoid applied for interpretation, and multiclass classification commonly produces logits with softmax used for probabilities. The loss function should decide whether the activation belongs inside the model.

Say this in the interview: A stack of affine layers is still one affine map, so activations are what give a neural network nonlinear boundaries and feature interactions; the hidden activation is essential for expressivity, while the output activation depends on the task and loss.