Support vector machines

Many lines can separate two classes. A support vector machine (SVM) asks a sharper question: which boundary leaves the widest gap between the classes? That single idea — the maximum margin — gives SVMs strong generalization, and the kernel trick that comes with it is one of the most elegant moves in all of ML (and a perennial interview question).

Max margin: the widest street

Picture the boundary as a street between the two classes. The SVM makes that street as wide as possible. The points that touch the curb — the closest ones on each side — are the support vectors, and they’re the only points that define the boundary. Move a far-away point and nothing changes; move a support vector and the whole boundary shifts. That focus on the hardest cases is why SVMs generalize well.

Soft margins and C

Real data isn’t cleanly separable, so SVMs use a soft margin — they allow some points to sit inside the margin or on the wrong side, for a penalty. The C hyperparameter sets how much you punish those violations:

• Low C → a wider, more tolerant margin (more regularization). Accepts some misclassifications for a smoother boundary. Higher bias, lower variance.
• High C → a narrow margin that tries hard to classify every training point correctly. Lower bias, higher variance — risks overfitting.

C is the SVM’s bias–variance dial: low C widens the street and tolerates errors; high C narrows it to fit every training point.

The kernel trick — bending the boundary

A straight line can’t separate concentric circles. The SVM’s superpower is the kernel trick: it implicitly maps the data into a higher-dimensional space where a straight boundary does exist, without ever computing those coordinates — it only needs dot products, which the kernel computes directly.

• Linear kernel — a straight boundary. Fast; the right choice for high-dimensional data (text) that’s already roughly linearly separable.
• RBF (Gaussian) kernel — the default for non-linear data; wraps curved, blobby boundaries. Tuned by gamma (how local each point’s influence is).
• Polynomial — curved boundaries of a fixed degree.

import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_circles

# Concentric circles — NOT linearly separable.
X, y = make_circles(n_samples=500, noise=0.08, factor=0.4, random_state=0)

for kernel in ["linear", "rbf"]:
    # SVMs need scaled features (they compare distances).
    clf = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=1.0))
    acc = cross_val_score(clf, X, y, cv=5).mean()
    print(f"{kernel:>6} kernel: {acc:.3f} CV accuracy")

print("\nLinear can't wrap a ring (~50%); RBF separates it (~99%).")

In one breath

• An SVM picks the boundary with the widest margin; only the support vectors (the closest points, on the margin) define it — move a far point and nothing changes.
• Soft margins allow some violations, and C is the bias-variance dial: low C = wider, tolerant margin (more bias); high C = narrow margin that fits hard (more variance, overfit risk).
• The kernel trick maps data into a higher-dimensional space where a linear boundary exists, using only dot products — linear (high-D/text), RBF (curved, tune gamma), polynomial.
• RBF is not a free upgrade — match the kernel to the data (start linear); RBF’s two knobs (C and gamma) overfit easily.
• Always scale features (SVMs compare distances). Best on high-dimensional, smaller datasets; gradient-boosted trees usually win on large tabular data.

Quick check

Next

The last classification topic before the tree ensembles: every real problem is lopsided, so class imbalance — and the fixes that stop a model from just predicting the majority.