What does the C parameter control in a Support Vector Machine?
C is the regularisation parameter that trades margin width against training error tolerance. A small C allows many margin violations (wide margin, simpler boundary, higher bias) while a large C penalises violations heavily, forcing a narrow margin that fits the training data more tightly but risks overfitting.
How to think about it
C is the most important hyperparameter for SVMs and appears in nearly every interview on the topic. Link it to the bias-variance tradeoff rather than describing it mechanically.
The soft-margin objective
Hard-margin SVM requires perfect separability. Soft-margin SVM introduces slack variables ξ_i ≥ 0 that allow a point to violate the margin:
Minimise: ½ ‖w‖² + C · Σ ξ_i
subject to: y_i(w · x_i + b) ≥ 1 - ξ_i
The two terms are in direct tension:
- ½ ‖w‖²: wants a wide margin (small ‖w‖), which means a simpler decision boundary.
- C · Σ ξ_i: penalises every margin violation by amount C.
Increasing C makes the model pay more for violations, so it shrinks the margin to accommodate the training points — higher variance. Decreasing C accepts more violations in exchange for a wider, more robust margin — higher bias.
Practical effect
| C value | Margin | Training error | Generalisation risk |
|---|---|---|---|
| Very small (0.001) | Wide | High | Underfitting |
| ~1 (default) | Balanced | Moderate | Often good starting point |
| Very large (1000+) | Narrow | Low | Overfitting |
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
import numpy as np
# Log-uniform search over C — linear spacing misses orders of magnitude
for C in np.logspace(-3, 3, 13):
score = cross_val_score(SVC(C=C, kernel="rbf", gamma="scale"),
X_train, y_train, cv=5).mean()
print(f"C={C:.4f} CV accuracy={score:.4f}")Always search C on a log scale — the relevant range spans several orders of magnitude, and linear spacing wastes budget.