Which models require feature scaling and which don't, and why?

How to think about it

The crisp answer

Scale features for algorithms that compare distances, variances, or gradient steps: KNN, K-means, SVM, PCA, regularized linear and logistic regression, and neural networks. Tree-based models generally do not need scaling because their splits depend on feature ordering, not on the numerical size of a feature.

There is one important qualification: ordinary unregularized linear or logistic regression can produce the same predictions with or without scaling, assuming exact optimization and no numerical problems. Scaling is still usually helpful in practice.

What feature scaling actually does

Feature scaling means transforming numeric columns so their values use comparable units. Two common transformations are:

• Standardization: z = (x - mean) / standard_deviation. The feature has approximately mean zero and standard deviation one.
• Min-max scaling: x_scaled = (x - minimum) / (maximum - minimum). Training values are mapped into [0, 1].

The scaler learns those statistics from the training data. It then applies the same statistics to validation, test, and production data.

That last detail matters. If the average income in the test set is used to scale the training rows, information from the test set has slipped into training. The model has not seen the labels, but it has still seen part of the future data distribution. That is data leakage.

The central question is simple: does the algorithm care about the coordinate system?

If it measures a distance, maximizes variance, or takes gradient steps through a weighted sum, the answer is usually yes. If it asks whether one value is before or after a threshold, the answer is usually no.

Distance-based models: the large unit wins

Suppose a KNN classifier predicts whether a customer will churn. Its two features are age and annual income.

The query customer is:

Q = (age 26, income 41000)

Two possible neighbors are:

• N1 = (age 27, income 45000)
• N2 = (age 50, income 40100)

Using ordinary Euclidean distance on the raw values:

• Distance from Q to N1 is approximately 4000.00.
• Distance from Q to N2 is approximately 900.32.

KNN calls N2 the closer customer. That may be a poor business interpretation: the second customer is 24 years older, but income is measured in dollars, so the age difference barely counts.

Now suppose the training ranges are age 20 to 70 and income 20,000 to 120,000. Min-max scaling gives:

• Q = (0.12, 0.21)
• N1 = (0.14, 0.25)
• N2 = (0.60, 0.201)

The scaled distances are approximately:

• Q to N1: 0.0447
• Q to N2: 0.4801

After scaling, N1 is clearly closer. The algorithm has not become smarter. We have stopped allowing dollars to drown out years.

K-means has the same problem because it assigns points to the nearest centroid. A feature measured in milliseconds can dominate a feature measured in percentages, even when the percentage is more important to the problem.

SVMs also need careful scaling. An RBF kernel uses a relationship of the form exp(-gamma * squared_distance). Multiply one feature by 100 and its contribution to squared distance becomes 10,000 times larger. The same gamma now describes a completely different geometry. Linear SVMs are affected too, because the margin, coefficient penalty, and optimizer all depend on feature units.

This is why a model with an excellent choice of C and gamma can still perform badly if the inputs are on wildly different scales.

PCA: variance has units too

PCA, or principal component analysis, finds directions that explain the most variance. Variance is measured in squared units.

Imagine a dataset with height in metres and income in dollars. A small change in income can produce much more numerical variance than a large relative change in height. PCA may therefore devote its first component almost entirely to income. That is not a bug in PCA. It is faithfully answering the question “which direction has the largest numerical variance?”

If the variables should contribute equally by their relative variation, standardize them first. Conceptually, this means running PCA on the correlation structure rather than allowing raw measurement units to decide the result.

But standardizing PCA is not automatically correct. If the units and absolute variance are meaningful, removing that information may be wrong. For example, in a manufacturing process, a high-variance sensor may genuinely be the most important source of variation. The scaling decision is a modelling decision, not a ritual.

Gradient-based models: optimization and regularization

For a linear or logistic model, scaling affects the optimization landscape.

Suppose a logistic regression uses age in years and income in dollars. A plausible coefficient might be:

• Age coefficient: 0.20 per year
• Income coefficient: 0.00001 per dollar

Both coefficients can represent meaningful effects. But with L2 regularization, the penalty includes terms such as lambda * w_age^2 and lambda * w_income^2. The age term contributes 0.04; the income term contributes only 0.0000000001. The regularizer penalizes the age coefficient much more heavily simply because age uses smaller numerical units.

After standardization, both coefficients describe the effect of a one-standard-deviation change. The penalty can then compare them more fairly.

Scaling also improves the shape of the optimization problem. The gradient with respect to a feature’s coefficient is influenced by that feature’s magnitude. If one column ranges from 0 to 1 and another from 0 to 100,000, a single learning rate is a poor compromise: small steps are painfully slow for one direction, while large steps may overshoot in the other. Scaling often makes the contours less elongated, so gradient descent can move toward a solution instead of zig-zagging down a narrow valley.

For unregularized ordinary least squares, scaling is not mathematically required for the fitted predictions. The weights can compensate for a change of units. In practice, scaling can still improve numerical conditioning and optimizer convergence. Once L1, L2, elastic-net penalties, early stopping, or a shared optimization procedure enters the picture, scaling becomes much more important.

Neural networks usually benefit from inputs that are roughly centred and similarly scaled. Large inputs can push sigmoid or tanh units into saturation, where their gradients become tiny. Uneven input scales also make the first layers harder to optimize. ReLU networks do not have sigmoid’s exact saturation problem on the positive side, but their activations, initial weights, and gradient sizes are still affected by input magnitude.

Why trees generally do not care

A decision tree asks questions such as:

income > 50000

If income is min-max scaled using a training range of 20,000 to 120,000, the same split becomes:

income_scaled > 0.30

Every row that was on one side of the original threshold remains on that side of the new threshold. The threshold value changes, but the ordering does not.

That is why decision trees, random forests, and most gradient-boosted tree implementations such as XGBoost and LightGBM generally do not need feature scaling. They split one feature at a time, and scaling by a monotonic transformation preserves the order needed to find the split.

The word “generally” earns its keep here. Histogram-based implementations, floating-point rounding, and approximate split algorithms can produce tiny numerical differences after transformation. A nonlinear transformation can also interact with missing-value handling or implementation details. Those effects are usually much smaller than the distance distortion caused by skipping scaling for KNN or an RBF SVM.

A production-safe pattern

For an all-numeric KNN model, put the scaler and estimator in one pipeline:

from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(
    StandardScaler(),
    KNeighborsClassifier(n_neighbors=5)
)

model.fit(X_train, y_train)
predictions = model.predict(X_test)

The scaler is fitted when model.fit runs. During prediction, it only transforms the new rows using the training mean and standard deviation.

The pipeline is especially important during cross-validation. Without it, a common mistake is to scale the entire dataset first and then split it into folds. With a pipeline, each training fold learns its own scaling statistics, and its validation fold is transformed without contributing statistics.

For mixed data, scale numeric columns separately and handle categorical columns with an appropriate encoder. Do not scale an identifier merely because it is stored as an integer. An account number of 9002 is not “larger” in a meaningful feature sense than account number 17.

Be careful with sparse one-hot data as well. Centring a sparse matrix can turn most of its zeros into nonzero values and cause a sudden memory explosion. A practical pipeline may scale sparse numeric features without centring them, or leave one-hot columns as they are after deciding how much influence they should have on the model’s geometry.

The nuance: scaling is not always the right first move

Standardization is sensitive to outliers because the mean and standard deviation are sensitive to outliers. If income contains one billion-dollar record, many ordinary incomes may be compressed together. A robust scaler based on the median and interquartile range may be more appropriate.

Min-max scaling does not make future values stay inside [0, 1]. A production income above the training maximum can map to a value greater than one. That is not necessarily an error, but the model and any downstream assumptions must tolerate it.

Also distinguish feature scaling from vector normalization. Standardization changes each column using its population statistics. L2 normalization rescales each individual row to have unit length. For text embeddings and cosine-style retrieval, row normalization may be the relevant operation; standardizing every embedding dimension can destroy the geometry the embedding model was trained to provide.

Finally, scaling tree inputs is harmless in many workflows but not useful by itself. If the next step is a tree model, the extra preprocessing adds a fitted artefact, a possible train-serving mismatch, and another thing to monitor without usually improving the split logic.

What they’ll ask next

“Should I use standardization or min-max scaling?”

Use standardization as the default for KNN, SVM, PCA, and regularized linear models because it handles ordinary unbounded numeric features without requiring known limits. Use min-max scaling when bounded inputs are useful or required by a downstream method. If outliers are severe, consider a robust transformation. The data distribution and model objective matter more than the name of the scaler.

“Does scaling change SVM hyperparameters?”

Yes. gamma is tied to squared distances, so changing feature units changes the useful range of gamma. Scaling also changes the effective role of C through the margin and regularization geometry. Scale first, then tune the SVM; do not carry hyperparameters from the unscaled version and assume they mean the same thing.

“What happens if I fit the scaler before cross-validation?”

Validation statistics leak into the training fold. The score may look slightly or substantially better than the score you will get on genuinely future data. Fit the scaler inside a pipeline so every fold learns its transformation from its training portion only.

Say this in the interview

“Scale models whose geometry, variance, regularization, or optimization depends on feature magnitude—such as KNN, K-means, SVM, PCA, and neural networks—while trees usually do not need it because threshold splits preserve ordering; in production I fit the scaler on training data only and apply it unchanged everywhere else.”