Support vector machines

It is 3 a.m. A new batch of sensor readings arrives, and your classifier must decide which machine parts are defective. The data is not a neat textbook split. A few good parts look suspicious. One defective part sits among the good ones. Several possible lines separate most of the points.

Which line should go into production?

A support vector machine (SVM) chooses the separating boundary with the largest possible margin: the widest buffer between the boundary and the nearest examples from either class. That sounds like a small geometric preference. It is actually a complete modelling strategy.

The margin makes the model reluctant to react to harmless, far-away observations. In the hard-margin picture, support vectors lie on the margin. In a soft-margin SVM, any point on or inside the margin, or on the wrong side, may be a support vector.

A soft margin allows reality to be messy. A kernel lets a boundary curve when a straight one cannot do the job.

SVMs are not the default for every dataset in 2026. On a large tabular dataset, gradient-boosted trees often win with less ceremony. But on a small, wide dataset such as TF-IDF text features, an SVM remains an unusually strong tool. It is also one of the cleanest ways to understand what “regularization” means geometrically.

The widest street

Start with two classes that a straight line can separate. A linear classifier describes that line, or its higher-dimensional equivalent, with a score:

f(x) = w · x + b

The decision boundary is where the score is zero. One side gets class +1; the other gets class -1.

An SVM asks how far the nearest training point is from that boundary, measured perpendicularly, and makes that distance as large as possible. The region between the two nearest class examples is the margin. The examples that touch its edges are the support vectors.

A wider margin gives predictions room to absorb small measurement changes. It is therefore a geometric form of regularization: the model prefers a less sensitive separation.

This is not a guarantee. A wide margin in the wrong feature representation is still wrong; if the true pattern is a circle, a straight boundary is a regularized mistake.

The actual optimization

The equation has a scaling ambiguity. Multiplying both w and b by ten leaves the boundary in place while changing the scores. SVM training removes that ambiguity by placing the closest positive examples at +1 and the closest negative examples at -1.

With labels y_i equal to +1 or -1, a perfectly separable, or hard-margin, SVM solves:

minimize 1/2 ||w||²

subject to:

y_i (w · x_i + b) >= 1

The perpendicular distance from the boundary to either margin line is 1 / ||w||, so the full street is 2 / ||w||. Minimizing the squared length of w is exactly the same as maximizing that street.

The constraint explains the support vectors. A point exactly at y_i (w · x_i + b) = 1 or -1 holds the street open.

A point comfortably beyond its margin satisfies the constraint with room to spare and does not push the optimum.

The dual form makes this precise. It assigns a coefficient alpha_i to each training point, producing:

f(x) = sum(alpha_i y_i K(x_i, x)) + b

Only points with non-zero coefficients appear in that sum. Those are the support vectors.

For a linear kernel, K(x_i, x) is the ordinary dot product; for a nonlinear kernel, it is a similarity function. Other points were considered during training, then found too far away to constrain the final boundary.

A concrete example

Here is the geometry with real numbers. Suppose positive points are (2, 2) and (2, -2), while negative points are (-2, 2) and (-2, -2). The natural boundary is x_1 = 0.

Choosing w = (0, 0.5) is wrong: it uses x_2, the coordinate that does not separate the classes.

With w = (0.5, 0) and b = 0, the positive points have score +1 and the negative points have score -1. Every point is on a margin, and the distance from the boundary to either margin is 1 / 0.5 = 2, making the full margin 4.

Add positive points at (10, 10) and (10, -10). They still satisfy the constraints, but moving one from x_1 = 10 to x_1 = 11 does not change the boundary.

Move a support vector from (2, 2) to (1.5, 2), and the widest feasible street must change. The optimization is controlled by the nearest constraints, which is why only support vectors have non-zero influence in the final solution.

Soft margins: what C really buys

Clean separation is a classroom luxury. A label may be wrong, a sensor may glitch, or the classes may genuinely overlap.

A soft-margin SVM permits a point to enter the street or cross the boundary. Each violation gets a non-negative slack variable xi_i:

y_i (w · x_i + b) >= 1 - xi_i

A point with xi_i = 0 is correctly classified and outside the margin. A point with 0 < xi_i < 1 is correctly classified but inside it.

A point with xi_i > 1 is on the wrong side of the boundary.

The objective becomes:

minimize 1/2 ||w||² + C sum(xi_i)

subject to the constraint above and:

xi_i >= 0

The first term wants a wide street; the second charges for violations. C sets the exchange rate:

• Low C makes violations cheap, favouring a wider, smoother margin.
• High C makes violations expensive, pushing the boundary to classify training examples more aggressively.

Thus low C usually means more bias and less variance, while high C can reduce training error at the cost of greater variance. C does not allow exactly a certain number of errors; it prices the total margin violation.

A high C is dangerous around label errors: the model cannot know that a “defective” label was mistyped, so it bends the boundary to pay for the costly violation.

A lower C says that one troublesome example is not worth shrinking the street for everyone.

Scale features before tuning. If one feature is measured in dollars and another in millimetres, changing units changes the geometry and what a given C means.

Select C with cross-validation inside the training data, not by training accuracy, and keep the final test set untouched. See the train, test, and cross-validation workflow.

Kernels: a straight line in a better space

Some patterns cannot be separated by a straight line in the original features. With concentric circles, for example, any line cuts through both classes.

import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_circles

X, y = make_circles(n_samples=500, noise=0.08, factor=0.4, random_state=0)

for kernel in ["linear", "rbf"]:
    clf = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=1.0))
    accuracy = cross_val_score(clf, X, y, cv=5).mean()
    print(f"{kernel:>6} kernel: {accuracy:.3f} CV accuracy")

A kernel defines an implicit feature space where the pattern may become linearly separable. It computes inner products in that space without materializing every new coordinate; it does not guarantee a useful boundary.

For a point (x_1, x_2), adding the feature x_1² + x_2² turns a circle into a threshold on one feature, so a line in the new space appears as a circle in the original plot.

Common choices are:

• Linear: K(a, b) = a · b. Use it first for sparse text and other high-dimensional data.
• Polynomial: creates interactions of a chosen degree, but degree and scaling can easily make it too flexible.
• RBF, or radial basis function: K(a, b) = exp(-gamma ||a - b||²). Nearby points are similar, while distant points contribute less, allowing curved local boundaries.

gamma controls how quickly RBF similarity falls with distance. Low gamma gives broad, smooth regions and can underfit.

High gamma gives narrow, local regions and can draw small islands around training examples.

C and gamma interact. High C makes training violations expensive; high gamma makes each point influential only nearby. Together they can memorize a small dataset.

Tune them jointly on logarithmic grids, such as C in 0.01, 0.1, 1, 10, 100 and gamma in 0.001, 0.01, 0.1, 1.

A kernel cannot rescue a feature set with no useful signal. Its notion of distance is only as sensible as the units and representation you provide.

For sparse TF-IDF matrices, do not blindly centre the matrix: subtracting a mean can destroy sparsity. Use preprocessing suited to the representation and keep it inside the validation pipeline.

What to use in practice

Use a linear SVM for high-dimensional, reasonably represented data such as TF-IDF text, one-hot categories, or a wide-but-short biological matrix.

Use an RBF SVM when the dataset is small or medium-sized, features are dense and scaled, and validation shows that local curvature matters.

For large tables with mixed types, threshold-like interactions, missing values, or irregular scales, compare a tree ensemble; it often scales better and needs less preprocessing.

A kernel SVM compares new points with its support vectors, and its general kernel training can require a pairwise matrix whose storage grows roughly quadratically with the number of training examples.

For very large datasets, try a linear SVM (LinearSVC in scikit-learn) or gradient boosting instead.

An SVM normally returns a decision score, not a probability. A score of 2 does not mean “twice as likely” as 1.

If a downstream system needs calibrated statements such as “there is a 70% chance this part is defective,” calibrate scores on data separate from the fitting data; see model calibration.

A practical workflow is:

1. Define the operational split and error metric.
2. Put scaling and other learned preprocessing in a pipeline.
3. Establish a linear baseline.
4. Try an RBF kernel only when validation suggests a straight boundary is inadequate.
5. Tune C and gamma jointly inside the training data.
6. Inspect errors, minority-class behaviour, and calibration before evaluating once on the untouched test set.

Check the confusion matrix and recall as well as accuracy when defects are rare.

Class weights such as class_weight="balanced" can increase the penalty on the minority class; the class imbalance guide covers the trade-off.

For leakage, keep imputation, feature selection, and scaling inside the pipeline, and use time-aware splits when predictions follow a time order. See data leakage.

Quick check

Next

Real classification problems are often lopsided. Continue with class imbalance to stop a model from earning impressive accuracy by ignoring the minority class.