Why does R-squared always increase when you add features, and when should you use adjusted R-squared instead?

How to think about it

On the same training rows, ordinary least squares, or OLS, can only make R-squared, the proportion of target variation explained, rise or stay flat when a feature, an input column, is added to a nested model, a model containing all the old predictors plus the new one. Adjusted R-squared is useful for comparing such linear models because it charges for each predictor, but held-out or cross-validated error is the better choice when the goal is prediction.

Why R-squared cannot fall

R-squared is usually written as:

R² = 1 - RSS / TSS

Here, RSS, the residual sum of squares, is the sum of squared prediction errors:

RSS = Σ(yᵢ - ŷᵢ)²

TSS, the total sum of squares, measures how much the target varies around its mean:

TSS = Σ(yᵢ - ȳ)²

For a fixed dataset, TSS is fixed. The target values have not changed, so their overall variation has not changed.

OLS chooses the coefficients that produce the smallest possible RSS. Now suppose the original model predicts house prices from square footage and number of bedrooms. Add a third predictor: a random number generated from a computer.

The new model can reproduce every prediction made by the old model. It simply gives the random column a coefficient of zero. The old model is therefore one valid setting of the new model. Since OLS searches over at least all the old settings, and possibly more, its best RSS cannot be larger.

That is the entire guarantee:

• the old model is still available;
• the new model has extra coefficient choices;
• OLS picks the choice with the lowest training RSS;
• TSS stays fixed;
• therefore R-squared cannot decrease.

It may stay exactly the same. Do not say that OLS must assign a tiny but nonzero coefficient to every new feature. A coefficient can be exactly zero, or the new feature may fail to improve the fitted predictions. The guarantee is non-increase in training error, not a guaranteed increase.

A random feature often does improve training fit anyway. OLS does not know that a column is meaningless. If the random values happen to line up with the current residuals, the model can use that accidental pattern to shave a few points off RSS. It has found correlation, not useful signal. The distinction tends to become painfully clear when the model meets new data.

A numerical example

Imagine 100 property sales. We fit a linear model using square footage and bedrooms, so there are n = 100 observations and p = 2 predictors. Suppose the target variation is:

TSS = 1,000

The two-predictor model has:

RSS = 400

Its training R-squared is:

R² = 1 - 400 / 1,000 = 0.600

Now add a random column. The results could look like this:

Model	Predictors	RSS	R-squared	Adjusted R-squared
Square feet and bedrooms	2	400	0.600	0.592
Plus a weak random column	3	398	0.602	0.590
Plus a lucky random column	3	390	0.610	0.598

In the weak case, R-squared rises from 0.600 to 0.602. That looks like progress if you inspect R-squared alone. Adjusted R-squared falls from about 0.592 to 0.590, because the two-point reduction in training error is not enough to justify estimating one more coefficient.

In the lucky case, the random column removes 10 points of residual error. That is enough for adjusted R-squared to rise to about 0.598.

The same principle explains why adding dozens of irrelevant columns can produce an impressive training R-squared. With enough chances to fit accidental patterns, noise gets its little lottery ticket.

What adjusted R-squared changes

Adjusted R-squared is:

R²_adj = 1 - (1 - R²) × (n - 1) / (n - p - 1)

n is the number of observations. p is the number of fitted predictor coefficients, normally excluding the intercept. The formula assumes the usual linear regression setup with an intercept and enough observations to estimate the coefficients.

The adjustment accounts for lost degrees of freedom. Degrees of freedom here means the independent information left after estimating model parameters. Every extra predictor consumes one of those degrees of freedom. The model therefore has fewer independent residuals with which to estimate its remaining unexplained error.

For the property example, the original model has n - p - 1 = 97 residual degrees of freedom. After adding one predictor, it has 96. Adjusted R-squared asks whether the reduction in residual error is large enough to compensate for that loss.

For one added predictor, adjusted R-squared increases only when the new residual error, after accounting for its smaller degrees-of-freedom denominator, is better. In formula form, the condition is:

RSS_new / (n - p - 2) < RSS_old / (n - p - 1)

For the original model, the extra feature needs to reduce RSS by more than approximately:

400 / 97 ≈ 4.12

The weak random column reduces RSS by only 2, so adjusted R-squared falls. The lucky one reduces it by 10, so adjusted R-squared rises.

This is why adjusted R-squared is not a fixed penalty such as “subtract 0.01 for every feature.” The penalty depends on both the sample size and the number of predictors. Adding one feature to a dataset with 10,000 observations costs much less than adding one feature to a dataset with 30 observations and 25 existing predictors.

The word “always” is doing too much work

The monotonic result applies to training predictions from nested, unregularized OLS models fitted on the same observations.

The word “training” matters most. On held-out data, R-squared can go down after adding a feature because the extra feature may describe quirks of the training sample rather than a reusable relationship. A model can have training R-squared of 0.61 and test R-squared of 0.48. The extra feature improved the data it saw and damaged its performance on data it did not see.

Test-set R-squared can even be negative. That means the model predicts worse than a simple baseline that predicts the relevant mean target for every test observation. R-squared is not guaranteed to stay between zero and one outside the training fit.

The guarantee also changes when the models are not genuinely nested. If the candidate model uses a different set of rows because of missing values, or if preprocessing changes the data, the two TSS values may differ. You are no longer comparing the same problem.

Regularized models such as ridge and lasso add a penalty for large coefficients to the fitting objective. Their training objective is not simply “minimize RSS.” A larger feature set can therefore produce a different trade-off between residual error and coefficient penalty. The OLS proof does not automatically apply to the raw residual error from those fits.

When should you use adjusted R-squared?

Use adjusted R-squared when comparing competing linear-model specifications on the same response, same observations, and roughly the same modeling assumptions. It is useful when you want a compact model and need a quick in-sample measure that does not reward every extra column for merely existing.

It is especially reasonable for an explanatory analysis. For example, you might compare whether a regression explaining delivery time should include three operational predictors or eight, and report adjusted R-squared alongside residual diagnostics and coefficient uncertainty.

Do not use it as a universal feature-selection command.

If the purpose is prediction, use cross-validation. Cross-validation repeatedly fits the model on one portion of the training data and evaluates it on a different portion that was not used for that fit. Compare a metric that matches the business cost, such as root mean squared error when large misses are especially painful, or mean absolute error when a linear penalty is more appropriate. Keep a final test set untouched until the end.

If a feature is needed to control for a confounder, do not remove it merely because adjusted R-squared falls. A confounder is a variable related to both the predictor and the outcome. Omitting it can distort the coefficient you care about, even if the omitted variable contributes little to predictive fit. Model purpose comes before a single score.

Akaike information criterion and Bayesian information criterion, commonly called AIC and BIC, are other complexity-adjusted criteria. They are based on a model likelihood and use different penalties. They can be useful for statistical model comparison, but neither is a substitute for out-of-sample validation when deployment accuracy is the goal.

One practical workflow is simple: fit candidate models on identical training rows, inspect adjusted R-squared if you need an in-sample parsimony comparison, use cross-validation for model selection, and evaluate the final locked model once on held-out data.

What they’ll ask next

Does adding a random feature always increase R-squared?

It always leaves training R-squared unchanged or higher under nested OLS. It does not have to increase. If the new column provides no additional direction for reducing residual error, OLS can set its coefficient to zero. On unseen data, R-squared may increase, stay flat, or fall.

How do you count predictors in adjusted R-squared?

Count fitted coefficient columns, not business concepts. With an intercept and reference coding, a categorical variable with five levels usually contributes four dummy-variable coefficients. An interaction or spline can contribute several coefficients as well. Those columns all consume degrees of freedom. A duplicated or perfectly collinear column may create a rank-deficient design, so the effective parameter count needs care.

Is adjusted R-squared enough to choose the final model?

No. It is an in-sample criterion with a particular penalty, not a guarantee of generalization, statistical significance, or causality. I would combine it with cross-validated error, residual checks, domain reasoning, and a final held-out evaluation. I would also keep a scientifically necessary control variable even if it slightly lowers adjusted R-squared.

Say this in the interview

“For nested ordinary least-squares models on the same training rows, adding a feature cannot increase residual error because the old model remains available with the new coefficient set to zero, so R-squared cannot fall; adjusted R-squared charges for the lost degrees of freedom, while cross-validated error is better for judging prediction.”