Perceptron & the Update Rule

Last lesson asked for the simplest unit that learns in the plainest sense — watches its own mistakes and corrects. Here it is, and it happens to be the original artificial neuron, the ancestor of every neural network. Its prediction is brutally simple: compute the linear score z = wᵀx + b, then output its sign. If z is positive, predict +1; if negative, predict −1.

Unpack the notation before trusting it. wᵀx says nothing more than “multiply each input feature by its own weight and add the results up”. The b — the bias — is a single constant that shifts that total up or down. The boundary z = 0 is a line, just as in logistic regression, but instead of a smooth probability the perceptron commits to a hard ±1.

What made it historic is how it learns:

• no calculus,
• no probability,
• no loss to descend.

Instead, there is just a tiny correction applied every time it gets a point wrong. It repeats until it stops making mistakes — the way you twitch a steering wheel back each time the car drifts off its lane.

Predict with a sign, learn from mistakes

The prediction:

ŷ = sign(wᵀx + b)   →   +1 if wᵀx + b > 0,   −1 otherwise

The learning rule walks through the training points. When the prediction ŷ matches the true label y, do nothing. When it is wrong, nudge the weights:

The key is the error term (y − ŷ). When the prediction is right, y − ŷ = 0 and the weights do not move. When it is wrong — say y = +1 but ŷ = −1 — y − ŷ = +2, so we add a multiple of x to w. That pushes the score wᵀx up for this exact point, dragging it toward the positive side. Geometrically, each update rotates the decision boundary toward correctly classifying the point it just missed.

This repeats over the data, pass after pass. The Perceptron Convergence Theorem guarantees the process halts with zero errors. This guarantee holds only if the classes are linearly separable. That means some straight line (a flat hyperplane, in more than two dimensions) can be drawn with every positive point on one side of it and every negative point on the other. If no such line exists, the perceptron never settles; some point is always wrong, so the weights never stop moving.

That guarantee promises less than it appears to, and the gap is worth naming: converging is not the same as finding the best line. The perceptron stops the instant it makes zero mistakes on the training data. It therefore hands back whichever separating line it happened to stumble into — quite possibly one passing a hair’s breadth from a training point. Visit the points in a different order and you converge to a different line. Nothing in the algorithm prefers one over the other. “Halted” means no errors left, not widest gap found.

How GATE asks this

Almost always an MCQ or NAT asking for the effect of a single update. The prompt gives you:

• w and b,
• a misclassified point x and its true label y,
• the learning rate η.

Compute the new weights or show that the score wᵀx moves toward the correct side. This single-neuron update is also the building block behind the neural-network questions GATE DA has asked every year (2024–2026). A conceptual variant asks why a single-layer perceptron cannot learn XOR — the answer being that XOR is not linearly separable.

Worked example — one update step

Current weights w = (1, 0) with bias b = −3. A point x = (2, 1) has true label y = +1. With learning rate η = 1, perform one update and check that wᵀx improves.

First confirm it is a mistake. The score is z = wᵀx + b = (1)(2) + (0)(1) − 3 = 2 − 3 = −1. It is negative, so the perceptron predicts ŷ = −1 — wrong, since the true label is +1. Apply the update. The error term is y − ŷ = +1 − (−1) = 2:

w_new = w + η·(y − ŷ)·x
      = (1, 0) + 1 · 2 · (2, 1)
      = (1, 0) + (4, 2)
      = (5, 2)

Now recompute the weighted score wᵀx for the same point with the new weights:

old wᵀx = (1)(2) + (0)(1) = 2
new wᵀx = (5)(2) + (2)(1) = 10 + 2 = 12

The score jumped from 2 to 12 — far more strongly positive, exactly as the prediction prompt anticipated. The weight vector grew in the direction of x, which is what rotates the boundary toward the point it had been missing.

The bias moves by the same rule. It is the half of the update people forget: b_new = b + η(y − ŷ) = −3 + 1·2 = −1. Put both halves together and the full score for this point is now z = wᵀx + b = 12 − 1 = 11, comfortably positive — so the perceptron would label it +1 this time. One update turned the mistake into a correct call.

In one breath

The perceptron — the original neuron — predicts ŷ = sign(wᵀx + b), a hard ±1. It learns with no calculus by the rule w ← w + η(y − ŷ)x applied only to misclassified points (the error term y − ŷ is 0 when correct). Each correction rotates the boundary toward the point it missed. The Convergence Theorem guarantees it halts with zero errors iff the data is linearly separable. That condition is also its fatal limit: it can never learn XOR, because no single line splits it.

Practice

A question to carry forward

The perceptron learns — and then slams into a wall named XOR. Four points, two classes, and not one straight line on Earth can separate them. The single neuron loops forever, defeated by a problem a child solves at a glance.

The fix is the idea that launched deep learning: stop asking one neuron to do everything. Feed the outputs of several perceptrons into another perceptron. This creates a hidden layer between input and output, and suddenly curved, XOR-shaped regions come within reach. But the hard sign step has to go first: it is flat almost everywhere, so it tells a deeper network nothing about how to improve. Here is the thread onward:

• how does stacking neurons into layers buy the power to carve any region?
• what smooth activation must replace the sign so the stack can be trained?
• what does a multi-layer network actually compute as a signal flows through it?