Multi-Layer Perceptron & Activations

Last lesson left the lone perceptron defeated by XOR, with the cure already named: feed several neurons into another neuron, building a hidden layer. Do exactly that and you have a multi-layer perceptron (MLP) — the plainest neural network, and the architecture that ended the XOR impasse and launched deep learning. It is a stack of fully-connected layers, each one taking the previous layer’s outputs, mixing them with a weight matrix, adding a bias, and passing the result through a nonlinear activation. Logistic regression is just the one-layer special case; the MLP stacks more.

But the perceptron’s other parting warning was sharp: the hard sign step has to go. And there is a deeper reason than its flat gradient. Stack two linear layers with nothing between them and W₂(W₁x) collapses into a single linear map (W₂W₁)x — no more powerful than one layer, XOR still unsolved. The nonlinear activation between layers is what stops the collapse, and it is the smooth heir to the perceptron’s sign. With it, an MLP can bend boundaries into any shape; without it, depth buys nothing. Every deep network you will train — a tabular classifier, the dense blocks inside a transformer — is built from these fully-connected-plus-activation pairs, so counting their parameters is the first thing you do when sizing a model to fit in memory.

Layers, weights, and biases

A layer that maps a inputs to b units has a weight for every input-to-unit connection — a·b of them — plus one bias per unit, so b biases.

Total trainable parameters is the sum over layers. Per layer:

• with bias: a·b + b
• without bias: a·b

The classic activation choices: sigmoid (squashes to (0, 1)), tanh (squashes to (−1, 1)), and ReLU = max(0, x) — cheap to compute and largely free of the vanishing-gradient problem, but continuous everywhere yet NOT differentiable at x = 0 (the kink). Explore how each shapes its input and its gradient:

How GATE asks this

The signature question is a NAT: an architecture is given as a chain of layer sizes (e.g. 30 → 4 → 3 → 1) and you count the trainable parameters. The one thing that trips students is bias — the question states whether biases are included, and you must read it. MCQ/MSQ items test activation properties: ReLU’s non-differentiability at 0, and why a nonlinearity is needed at all.

Worked example — the no-bias and with-bias cases

Count the trainable parameters of two networks: 30 → 4 → 3 → 1 with no bias (a real GATE DA 2026 question), and 5 → 10 → 3 with bias.

(GATE DA 2026) Network 30 → 4 → 3 → 1, no bias. Multiply consecutive layer sizes and add:

weights = 30·4 + 4·3 + 3·1
        = 120  + 12  + 3
        = 135

So 135 trainable parameters.

Same idea, now 5 → 10 → 3 with bias. Each layer adds one bias per output unit:

layer 1 (5 → 10): 5·10 + 10 = 50 + 10 = 60
layer 2 (10 → 3): 10·3 +  3 = 30 +  3 = 33
total = 60 + 33 = 93

So 93 trainable parameters. The same logic, written once as a function:

def params(sizes, bias=True):
    total = 0
    for a, b in zip(sizes, sizes[1:]):     # consecutive layer sizes
        total += a * b + (b if bias else 0)
    return total

print("2026  30->4->3->1, no bias:", params([30, 4, 3, 1], bias=False))
print("with bias  5->10->3       :", params([5, 10, 3],    bias=True))

2026  30->4->3->1, no bias: 135
with bias  5->10->3       : 93

In one breath

A multi-layer perceptron stacks fully-connected layers — a layer from a inputs to b units holds a·b weights plus b biases — and inserts a nonlinear activation (sigmoid → (0,1), tanh → (−1,1), ReLU = max(0,x)) between them, without which the whole stack collapses to a single linear map; total trainable parameters are the sum over layers of a·b (+ b if biased), and the one activation fact GATE checks is that ReLU is continuous everywhere but not differentiable at 0.

Practice

A question to carry forward

So an MLP with a hidden layer and a smooth activation can carve XOR, and any region you like — and it can carry thousands of weights doing it. Which raises the obstacle that stalled neural networks for two decades: how do you train all those weights? Gradient descent needs ∂L/∂w for every single one, and the loss reaches each weight only after threading through layer upon layer of mixing and squashing.

Computing those gradients one weight at a time, from scratch, would be hopeless. But the chain rule from calculus suggests a shortcut — if you knew the gradient at a layer’s output, could you cheaply push it back to the layer’s inputs, and keep pushing, layer by layer, all the way to the first weight? Here is the thread onward: how does walking the chain rule backward through the network’s computation graph hand you every weight’s gradient in a single sweep — and what is the one local derivative the ReLU contributes as the gradient passes through it?