What is multicollinearity, how does it harm regression, and how do you detect and fix it?

How to think about it

The direct answer

Multicollinearity occurs when two or more predictors contain nearly the same linear information. Ordinary least squares can still produce accurate predictions, but it cannot reliably decide how much credit belongs to each correlated predictor, so coefficients become unstable, standard errors grow, and individual p-values become difficult to trust.

I detect it with feature inspection, pairwise correlations, Variance Inflation Factors, and condition diagnostics. I fix it by removing or combining redundant features, collecting data with more independent variation, or using regularization such as ridge regression when prediction matters more than interpreting each coefficient.

Why it harms regression

Suppose a regression predicts house price from living area and number of bedrooms. These features are not identical, but larger homes usually have more bedrooms. The model can estimate the combined effect of “a larger home with more rooms” quite well. It has a harder time separating the individual effect of one extra square foot from the individual effect of one extra bedroom.

In matrix notation, X is the feature matrix and β is the vector of coefficients. Ordinary least squares chooses β to minimise the squared prediction errors. The familiar normal-equation form is:

β̂ = (XᵀX)⁻¹Xᵀy

The equation exposes the problem. If one column of X is an almost-duplicate of another, then XᵀX is nearly singular, meaning it is difficult to invert accurately. The fitted predictions may barely change, but the individual coefficients can change dramatically.

With perfect multicollinearity, the problem is not merely instability. The coefficients are not uniquely identifiable. If x₂ = 2x₁, then many coefficient pairs produce the same prediction because the model only depends on the combination β₁ + 2β₂. There is no statistical evidence in that dataset for deciding how to split the effect between β₁ and β₂.

Near-perfect multicollinearity creates a very flat direction in the optimisation problem. Moving the coefficients along that direction changes predictions only a little, so ordinary sampling noise can move the fitted coefficients a long way. That is why coefficients may change sign when you add one related feature or refit on a new sample.

Under the usual fixed-design assumptions, the coefficient covariance is:

Var(β̂ | X) = σ²(XᵀX)⁻¹

Here, σ² is the residual noise variance. Multicollinearity makes parts of (XᵀX)⁻¹ large, which inflates coefficient variance and therefore standard errors. It does not, by itself, create bias in ordinary least squares. It creates imprecision.

Production libraries generally use QR or singular-value decompositions rather than explicitly calculating a matrix inverse. That is numerically safer, but it cannot manufacture information that the feature matrix does not contain.

A concrete numerical example

Consider 100 houses. Centre and standardise two predictors:

• size: living area
• rooms: number of rooms

Assume their Pearson correlation is r = 0.98. With only two predictors, the auxiliary regression of either feature on the other has:

R² = r² = 0.98² = 0.9604

The Variance Inflation Factor is therefore:

VIF = 1 / (1 - 0.9604) = 25.25

A VIF of 25.25 means the coefficient variance is 25.25 times the variance it would have with an orthogonal predictor design. The standard error is not 25.25 times larger; standard error is the square root of variance, so it is about √25.25 = 5.03 times larger.

Suppose the residual standard deviation is 10 units of the target and there are 100 observations. With an orthogonal predictor, the rough coefficient standard error would be:

σ / √n = 10 / √100 = 1

With this correlation, it becomes approximately:

1 × √25.25 = 5.03

If the target is measured in thousands of dollars, an estimated effect might carry a standard error of roughly $5,030 rather than $1,000. The exact number depends on the design and noise assumptions, but the mechanism is checkable: the unique information in each feature is small, so the coefficient estimate is noisy.

The model may still have an excellent R-squared. The house-price prediction depends largely on the combined “size and rooms” signal, not on the arbitrary division of that signal between the two columns.

How I detect it

I start with the feature definitions, not a threshold. Two columns named annual_revenue and monthly_revenue_times_12 are a data-modeling problem before they are a statistics problem. A domain review may reveal that a supposedly separate feature is a duplicate, a part-whole relationship, or a deterministic transformation.

Next, I inspect a correlation matrix and scatterplots for numeric features. Pairwise correlation is useful, but limited. It can find a strongly correlated pair; it cannot reliably find a feature that is predictable from a combination of several other features.

That is what VIF measures. For feature j, regress it on all the other predictors and call that auxiliary model’s R-squared R²ⱼ:

VIFⱼ = 1 / (1 - R²ⱼ)

A VIF of 1 means the feature has no linear relationship with the others. Values around 5 or 10 are commonly used as investigation points, not universal laws. A VIF of 10 means ten times the coefficient variance, and about 3.16 times the standard error. Whether that is acceptable depends on sample size, the purpose of the model, and how precise the coefficient must be.

A typical Python calculation looks like this:

import pandas as pd
import statsmodels.api as sm

from statsmodels.stats.outliers_influence import variance_inflation_factor

numeric_X = X.select_dtypes(include="number").dropna()
Z = sm.add_constant(numeric_X, has_constant="add")

vif = pd.DataFrame({
    "feature": numeric_X.columns,
    "VIF": [
        variance_inflation_factor(Z.to_numpy(), i)
        for i in range(1, Z.shape[1])
    ],
})

print(vif.sort_values("VIF", ascending=False))

The constant is included in the auxiliary regressions and excluded from the displayed results. Missing values must be handled consistently, and X should contain predictors rather than the target.

I also inspect the condition number or the singular values of a centred and standardised feature matrix. The condition number compares the largest singular value with the smallest one. A large ratio signals that some direction in feature space is poorly identified. Scaling matters here: a raw condition number can look alarming simply because one column is measured in dollars and another in years.

Finally, I compare coefficients across folds, bootstrap samples, or closely related model specifications. A practical symptom is a model whose R-squared and validation RMSE barely move while a coefficient jumps from positive to negative, its confidence interval becomes enormous, or its p-value changes from significant to insignificant. Those are classic signs that the model predicts the combined signal but cannot assign it cleanly.

Common misconception: high VIF does not automatically mean the whole regression is bad. It primarily damages individual coefficient precision. Prediction can also suffer when new observations break the correlation pattern seen during training, especially during extrapolation, but that is not guaranteed.

How I fix it

The correct fix depends on whether the model is for explanation or prediction.

Goal	First response	Main cost
Explain individual effects	Remove or combine redundant features	The scientific question may change
Predict accurately	Ridge or elastic net with validation	Coefficients become less direct
Reduce many correlated dimensions	PCA or another representation	Components are harder to explain
Obtain more identifiable effects	Collect data with independent variation	Often expensive or impossible

For inference, I first ask whether both features are genuinely needed. If living area and total rooms represent the same business concept, I may keep the one that matches the decision being made. I might also create a meaningful ratio or index, but only when its interpretation makes sense. Dropping a column because its VIF is highest is not a sufficient argument; the feature may be the one the business actually cares about.

Perfect multicollinearity also appears in one-hot encoding. If a categorical variable has three categories and the model includes an intercept plus all three dummy columns, the dummies sum to one. One category should usually be the reference category. Otherwise the design matrix contains an exact linear dependency.

For prediction, ridge regression is often the simplest reliable remedy. It minimises squared error plus an L2 penalty on the coefficients:

minimise ||y - Xβ||² + λ||β||²

The corresponding coefficient system uses XᵀX + λI. A positive λ prevents singularity in the penalised coefficient block and generally improves conditioning. Ridge accepts a small amount of bias in exchange for much lower variance. Correlated features tend to share the predictive weight instead of one coefficient exploding while another compensates.

I would standardise numeric features before ridge because the penalty depends on coefficient scale, then choose λ using cross-validation. The intercept is normally left unpenalised. Ridge stabilises prediction; it does not make the individual coefficients causally interpretable.

Lasso can set coefficients exactly to zero, which is useful for sparse selection, but with strongly correlated features it may choose one member of a group somewhat arbitrarily. Elastic net adds an L2 term and is often more stable when correlated groups should be retained.

PCA transforms the original predictors into orthogonal principal components. This removes linear correlation, but each component is a weighted combination of the original variables. PCA also chooses directions using predictor variance, not target relevance, so a high-variance direction is not automatically a useful predictive direction. Fit the PCA transformation on the training data only, then apply that fitted transformation to validation and test data.

More data helps only when it adds independent variation. If every record satisfies rooms = size / 500 by construction, collecting ten million more records will not identify separate room and size effects. The data collection process must include cases where one predictor changes without the other.

The senior nuance

Multicollinearity is a geometric problem, not the same thing as confounding. Confounding is a causal issue involving how variables relate to an outcome and to the data-generating process. A high VIF says that predictors overlap statistically; it does not say which variable causes anything.

The textbook statement that multicollinearity does not hurt predictions is directionally right but too broad. Predictions often remain stable for ordinary in-distribution cases because the unstable coefficient direction corresponds to a combination of features that rarely varies independently. Predictions can become unstable for unusual houses, new markets, or policy changes where the old relationship between size and rooms no longer holds.

I would also distinguish an individual coefficient test from a joint test. Several correlated variables can be jointly useful even when none has a small individual p-value. If the real question is whether the group contributes, test the group or define the estimable combined effect. Do not interpret a failed individual test as proof that every feature in the group is useless.

What they’ll ask next

Does high pairwise correlation always imply high VIF?
With two predictors, yes: VIF is 1 / (1 - r²). With many predictors, VIF measures how well one feature is explained by all the others, so several moderate correlations can produce a high VIF even when no pair looks extreme.

Does standardising the features fix multicollinearity?
No. Standardisation changes units and often improves numerical conditioning, but it does not remove the underlying linear relationship. A correlation of 0.98 remains 0.98 after scaling.

Would you choose ridge or PCA?
For prediction with the original features still useful, I would usually start with ridge because it preserves the feature space and gives a validation-tunable amount of shrinkage. I would use PCA when dimension reduction is itself valuable and component interpretability is acceptable. For coefficient inference, I would prefer a domain-informed feature redesign or more informative data.

Say this in the interview

“Multicollinearity is overlapping linear information among predictors: it inflates coefficient variance rather than necessarily hurting fit, so I diagnose it with VIF and condition checks, then remove or combine features or use ridge depending on whether interpretation or prediction is the goal.”