What do the C and gamma hyperparameters control in an SVM, and how do they relate to overfitting?

How to think about it

The direct answer

C controls how expensive margin violations are: large C pushes the SVM to classify training examples correctly, while small C accepts more violations in exchange for a wider, smoother margin. For an RBF SVM, gamma controls how local each training point’s influence is: large gamma allows a highly flexible boundary, while small gamma produces a smoother one. Large values of both often overfit, but neither parameter guarantees overfitting by itself.

Why C changes overfitting

An SVM does not merely search for any separating line. It searches for a boundary with the largest possible margin, meaning the distance between the boundary and the closest training points. Those closest points are called support vectors because they determine where the boundary sits.

Real data is noisy. A mislabeled customer, a faulty sensor reading, or two genuinely overlapping classes can make perfect separation undesirable. A soft-margin SVM therefore allows points to sit inside the margin or on the wrong side of the boundary. The variable representing that allowance is called slack.

The usual soft-margin objective looks like this:

minimize  1/2 * ||w||^2 + C * sum_i(xi_i)

subject to
          y_i * (w · phi(x_i) + b) >= 1 - xi_i
          xi_i >= 0

Here, y_i is the class label, usually -1 or +1; xi_i is the slack for example i; and w and b describe the boundary. phi represents the feature transformation used by a kernel.

The first term rewards a wide margin. The second term charges the model for violations. C controls the exchange rate between them.

• With small C, violations are cheap. The model prefers a smaller ||w||, which usually means a wider margin and a simpler boundary.
• With large C, violations are expensive. The model is willing to use a narrower margin or a more complicated boundary to reduce training mistakes.

For example, if one training point has xi = 2, its contribution to the objective is 0.2 when C = 0.1, but 200 when C = 100. The model has a much stronger reason to contort itself around that point in the second case.

C is therefore not the number of errors the model is allowed to make. It is a penalty applied to the size of the violations. A point just inside the margin and a badly misclassified point do not necessarily cost the same amount.

There is a useful dual interpretation too. Each training example receives a coefficient called alpha. That coefficient is capped at C; increasing C relaxes the cap and lets important or difficult points exert more influence. This is why changing C can change both the margin and which examples become support vectors.

In the usual statistical language, a larger C means weaker regularization, not stronger regularization. It places more weight on fitting the training data. That wording trips people up because “large penalty” sounds like “strong regularization.” The penalty is on mistakes, not on model complexity.

What gamma changes

gamma matters primarily for nonlinear kernels. The common RBF kernel, also called the Gaussian kernel, computes similarity between two examples using:

K(x, x') = exp(-gamma * ||x - x'||^2)

The value is near 1 when two points are close and approaches 0 as they move apart. gamma controls how quickly that similarity decays. It is effectively the inverse of the RBF width.

Assume the features have been standardized, so a distance of 1 means roughly one standard-deviation unit in the feature space.

With gamma = 0.1:

distance 1: exp(-0.1) = approximately 0.905
distance 3: exp(-0.9) = approximately 0.407

A point still has noticeable influence at a distance of 3.

With gamma = 10:

distance 1: exp(-10) = approximately 0.000045
distance 3: exp(-90) = approximately 0

Now a training point influences only extremely nearby examples. The model can create tiny local regions around individual observations. That gives it much more capacity to fit noise.

So:

• Low gamma means broad influence and a smooth, slowly changing boundary.
• High gamma means narrow influence and a boundary that can bend around individual points.

The RBF decision function is a weighted combination of these kernel similarities from the support vectors. Increasing gamma changes the entire similarity map, which changes the support vectors and, ultimately, the shape of the boundary.

For a polynomial kernel, gamma is still a coefficient that affects the kernel, but “radius of influence” is no longer the complete interpretation. For a linear kernel, gamma is ignored.

Warning: scale features before tuning either parameter. The RBF kernel uses squared distances, so units matter enormously. If one feature is annual income measured in dollars and another is age measured in years, the income feature can dominate the distance. A value of gamma that worked after standardization may be useless on raw data. Scaling also changes the effective meaning of C, because the geometry of the margin changes.

A concrete example

Imagine an SVM that flags suspicious bank transfers using two features:

• the transfer amount divided by the customer’s usual amount;
• minutes since the customer’s previous transfer.

Suppose there are 1,000 labeled transfers, both features are standardized, and 12 labels are noisy or genuinely ambiguous.

With C = 0.1, the model is allowed to leave some of those 12 points inside the margin if doing so preserves a broad boundary. With C = 100, each violation is much more expensive, so the model is pushed to isolate difficult points.

Now compare gamma = 0.01 with gamma = 10. At 0.01, a suspicious transfer can influence a fairly broad neighborhood of similar transfers. At 10, its influence is almost entirely local. Combining C = 100 and gamma = 10 gives the model both the ability and the incentive to create small pockets that classify the training set correctly. If those pockets describe labeling noise rather than fraud behavior, validation performance falls.

That is the relationship between the parameters: gamma controls the boundary’s available local complexity, while C controls how hard the model tries to use that complexity to eliminate training violations.

How the two parameters interact

This table gives the usual direction, not a law of nature:

C	gamma	Typical result
Low	Low	Very smooth boundary; likely underfitting
High	Low	Broad boundary that strongly pursues the overall pattern
Low	High	Local capacity exists, but mistakes may remain cheap
High	High	Highly local boundary; greatest risk of fitting noise

The important senior-level answer is that C and gamma must be tuned jointly. A high gamma does not automatically overfit if C is small enough that fitting every local irregularity is not worth the penalty. A high C does not automatically overfit if the data is clean and the classes are genuinely separable. Conversely, low values of both can underfit badly.

You can often see the problem in the scores:

• training and validation scores are both poor: the model is probably too constrained, often because both values are too low;
• training score is excellent but validation score is much worse: the model is probably too flexible, often because gamma is too high, C is too high, or both;
• scores vary wildly between folds: the dataset may be small, noisy, imbalanced, or sensitive to a few support vectors.

How I would tune them

I would scale inside the cross-validation pipeline, search logarithmically, and select a metric that matches the business cost. For a rare-fraud problem, accuracy is almost useless; average precision, recall at a required precision, or a cost-based metric is more informative.

A scikit-learn pattern could look like this:

from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

model = make_pipeline(
    StandardScaler(),
    SVC(kernel="rbf", class_weight="balanced")
)

search = GridSearchCV(
    model,
    {
        "svc__C": [0.01, 0.1, 1, 10, 100, 1000],
        "svc__gamma": [1e-4, 1e-3, 1e-2, 1e-1, 1],
    },
    scoring="average_precision",
    cv=StratifiedKFold(
        n_splits=5,
        shuffle=True,
        random_state=42,
    ),
    n_jobs=-1,
)

search.fit(X_train, y_train)

The grid is a starting range, not a universal answer. After scaling, it is sensible because the candidate values differ by powers of ten. If the best result sits at the edge of the grid, expand that edge rather than pretending the search found the optimum.

The test set should remain untouched until the model, preprocessing, and decision threshold are finalized. Otherwise the test set quietly becomes another training signal. For time-ordered data, ordinary shuffled cross-validation can also leak future patterns; use time-aware splits instead.

Failure mode and when not to use it

A common failure appears first as near-perfect training performance, a large validation gap, and a surprisingly high number of support vectors. The usual causes are an overly large gamma, an overly large C, noisy labels, or unscaled features. Lowering gamma often removes the tiny boundary “islands”; lowering C makes the model less willing to chase remaining exceptions. The fix should be confirmed with cross-validation, not chosen because the plot looks nicer.

The opposite failure is a model whose predictions are nearly constant. Training and validation scores are both low, and the decision boundary barely responds to the features. That usually points to values that are too small, or to a feature representation that does not separate the classes.

An RBF SVM is a strong choice for small or medium-sized tabular data with a genuinely nonlinear boundary. It is usually a poor first choice for millions of rows because kernel training becomes expensive as the number of examples grows, and prediction cost rises with the number of support vectors. A linear SVM is often better for very high-dimensional sparse text. Tree-based models or approximate kernel methods may be better when the dataset is large or contains many categorical features.

What they’ll ask next

Does a high C always overfit?

No. It increases the pressure to fit training examples, but overfitting depends on sample size, noise, feature scaling, kernel, and gamma. On clean, well-separated data, increasing C may improve validation performance or eventually stop changing the solution once all violations disappear.

Does gamma matter for a linear SVM?

No. A linear kernel has no RBF width, so gamma has no effect. For polynomial and sigmoid kernels, gamma affects the kernel coefficient, but its interpretation is different from the simple “local influence radius” explanation used for RBF.

How do you choose the best pair?

Put scaling and the SVM in one pipeline, search C and gamma on logarithmic ranges, use stratified or time-appropriate cross-validation, and score with the metric that reflects the real decision. Then evaluate once on an untouched test set.

Say this in the interview

“C controls how much the SVM penalizes margin violations, while RBF gamma controls how local each point’s influence is; high values can fit noise, so I scale features and tune both jointly with cross-validation.”