Why does regularization require feature scaling, and what happens if you skip it?

How to think about it

No—not mathematically. But with L1 or L2 regularization, you should usually scale continuous features first, because the penalty acts on coefficient values, and coefficient values change when the feature’s measurement unit changes.

Why the units change the penalty

A linear model predicts a target by adding a baseline to feature contributions:

prediction = intercept + beta_1 * x_1 + beta_2 * x_2 + ...

A coefficient, or beta, is the number multiplying a feature. Its size depends on the feature’s unit. A coefficient measured per dollar will naturally be much smaller than one measured per year.

Ordinary least squares, or OLS, chooses coefficients mainly by minimizing prediction error. Regularized regression adds another cost:

• L2 adds lambda * sum(beta_j ** 2).
• L1 adds lambda * sum(abs(beta_j)).

Here, lambda is the strength of the penalty. The exact scaling of the loss differs between libraries, but the important mechanism is the same: the regularizer sees coefficient magnitudes, not the real-world effect of changing a feature.

That is the problem. A coefficient of 0.0001 is not necessarily less important than a coefficient of 0.1. It may simply be attached to a feature whose values are 1,000 times larger.

Consider a model predicting an account-risk score. It uses:

• income, measured in dollars
• age, measured in years

Suppose a $10,000 increase in income raises the prediction by 1 point, and a 10-year increase in age also raises it by 1 point. Those are equally sized effects in the units that matter for this example.

The corresponding coefficients are:

• income: 1 / 10,000 = 0.0001
• age: 1 / 10 = 0.1

The unscaled L2 penalty sees:

• income: 0.0001 ** 2 = 0.00000001
• age: 0.1 ** 2 = 0.01

The age term contributes one million times more to the penalty. L1 has the same problem, although the ratio is 1,000 rather than one million:

• income: 0.0001
• age: 0.1

The regularizer has no idea that both coefficients represent the same practical effect. It only knows that 0.1 is larger than 0.0001.

Now express income in thousands of dollars instead. The same model uses:

• income: 0.1 per thousand dollars
• age: 0.1 per year

The predictions have not changed. The numbers in the penalty have changed dramatically.

That is the key test: if changing a feature from dollars to thousands changes which features Lasso selects or how strongly Ridge shrinks them, the model was reacting to units rather than evidence.

What happens if you skip scaling?

With L2 regularization, features with small numerical coefficients are penalized lightly. High-magnitude features such as income in dollars often fall into this category. Features with smaller numerical values, such as age, temperature, or a ratio, may need larger coefficients and therefore receive a heavier penalty.

Ridge can then shrink a low-scale feature more strongly than a high-scale feature, even when the low-scale feature has an equally large or larger effect on the prediction. L2 usually shrinks coefficients toward zero rather than making them exactly zero, so the symptom is distorted coefficient sizes and biased predictions, not necessarily a feature disappearing completely.

With L1 regularization, the problem is more visible. Lasso encourages exact zeros because the absolute-value penalty has a sharp corner at zero. If features are not on comparable scales, Lasso may set a small-scale feature to zero first simply because its coefficient is numerically expensive.

That does not mean Lasso has discovered that the feature is useless. It may have discovered that the feature is measured in inconvenient units.

Skipping scaling also makes the result dependent on arbitrary representation choices. A model trained with income in dollars can produce a different regularized solution from the same data trained with income in thousands of dollars. Unregularized OLS does not have this problem in theory: the coefficient changes to compensate, and predictions stay the same. Regularization breaks that unit invariance because it adds a cost to the coefficients themselves.

There is a second, separate effect: optimization. Gradient-based and coordinate-descent solvers behave better when features have similar scales. If one column ranges from 0 to 1 and another from 0 to 200,000, the loss surface becomes elongated. The optimizer may take steps that are sensible for one direction and poor for the other. The first symptoms can be slow training, sensitivity to the regularization value, unstable coefficients, or a convergence warning.

Scaling helps the geometry. It does not cure multicollinearity, bad features, or an incorrectly chosen penalty strength.

The standard production pattern

For continuous numeric features, z-score standardization is the usual default:

z_j = (x_j - mean_j) / standard_deviation_j

The mean and standard deviation are calculated separately for each feature. After transformation, a one-unit change means one training-set standard deviation.

A standardized coefficient of 2 means that increasing that feature by one standard deviation is associated with a 2-unit increase in the prediction, holding the other features fixed. This makes coefficient magnitudes comparable in a useful sense. It does not make them a complete measure of feature importance, especially when features are correlated.

In scikit-learn, put the scaler and model in one pipeline:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

model = Pipeline([
    ("scale", StandardScaler()),
    ("ridge", Ridge(alpha=1.0)),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

Pipeline.fit fits StandardScaler on X_train, transforms the training data, and then fits Ridge. At prediction time, it applies the same training mean and standard deviation to X_test.

Do not scale the training and test sets independently. If you do, a test value is expressed relative to the test distribution rather than the coordinate system the model learned. More importantly, using test statistics during preprocessing allows information from the evaluation set to influence the training procedure.

The same rule applies to cross-validation. If you tune alpha, the scaler must be inside the cross-validation pipeline so each fold learns its statistics from that fold’s training portion only. Otherwise, the validation fold quietly contributes to preprocessing. The score may look slightly better, which is exactly why this bug survives code review.

The intercept is normally not regularized by scikit-learn’s Ridge and Lasso implementations when fitting an intercept. Centering the features helps the intercept represent the prediction at the average feature values rather than carrying the burden of arbitrary feature offsets.

When scaling is not the right move

Scaling is important for penalized linear models, but it is not a ritual to perform on every dataset.

Decision trees split on thresholds. A tree can replace an income split at $80,000 with a split at 80 when income is represented in thousands. The ordering is unchanged, so Random Forest models generally do not need feature scaling. XGBoost-style tree models also generally do not need it.

Distance-based methods are a different story. K-nearest neighbors, clustering, and an RBF-kernel SVM can be dominated by a large-scale feature even without coefficient regularization, because distances and similarities are directly affected by feature magnitude. Scaling is often important there for a different reason.

Categorical and sparse features need deliberate handling. One-hot columns already have a natural zero-to-one scale, and standardizing them can change the meaning of their penalty. If the input is a sparse matrix, centering it can turn it into a huge dense matrix. A column-aware preprocessing design is safer; when using StandardScaler with sparse data, with_mean=False avoids densifying the matrix, though it also means the columns are not centered.

Outliers are another reason not to apply standardization blindly. One $10 million income value can inflate the standard deviation and compress ordinary incomes into a narrow range. A log transform, a robust scaler based on the median and interquartile range, or a business-specific transformation may be more appropriate.

There is also a modeling choice hidden inside “scale the features.” Standardization says that one standard deviation is a reasonable unit in which to compare penalties. That is often sensible, not universally correct. If domain knowledge says a particular feature should receive stronger or weaker shrinkage, encode that deliberately rather than relying on raw units to express the preference by accident.

A failure mode you can diagnose

Suppose a Lasso model trained on raw data keeps income and drops age. You convert income from dollars to thousands, retrain with the same alpha, and suddenly age remains while another feature disappears.

That unit-sensitive feature selection is the first clue. The model has not received new information. Only the coefficient scale changed.

A related symptom is that the solver needs far more iterations on the raw data, or reports that the objective did not converge, while the standardized version converges normally. Standardization often fixes the scale imbalance. If it does not, investigate correlated features, extreme outliers, an overly strong penalty, and the iteration or tolerance settings.

What they’ll ask next

Does scaling the target solve this problem?
No. The penalty is applied to feature coefficients, so feature scaling addresses the direct issue. Scaling the target can change the numerical meaning of alpha and may help some optimization procedures, but it is a separate decision.

Do I need to scale before every model?
No. Ridge, Lasso, Elastic Net, logistic regression, neural networks, SVMs, nearest-neighbor methods, and clustering commonly benefit from it. Tree-based models usually do not. The reason matters: scale features when the algorithm uses coefficient size, gradients, or distances in a scale-sensitive way.

How do I interpret coefficients after scaling?
A standardized coefficient describes the effect of a one-standard-deviation feature change. If you need the coefficient in the original unit, divide the standardized coefficient by that feature’s training standard deviation. Do not compare raw coefficient magnitudes across features unless their units and spreads are genuinely comparable.

Say this in the interview

“Regularization does not mathematically require scaling, but because L1 and L2 penalize coefficient values rather than real-world effects, unscaled features get unequal penalties based on their units; I standardize inside the training pipeline so the penalty and optimizer treat comparable variation comparably.”