Missing data and imputation

At 09:14, your loan model refuses to train.

The dataset has 48,000 applications. annual_income is blank in 11,600 rows. employment_length is blank in another 2,100. Scikit-learn prints:

ValueError: Input X contains NaN.

You delete incomplete rows. The error disappears, along with nearly a quarter of your customers. You replace missing income with zero. The model now treats “did not answer” as “earns nothing”. You replace it with the median and get a promising cross-validation score.

Then someone checks a fresh production slice. The score is worse.

Missing data is not just a nuisance between a CSV file and a model. A blank can mean a broken sensor, a skipped question, a business rule, or a person deliberately withholding an answer. Those causes change what a defensible replacement looks like.

The general name for replacing missing values is imputation. The important question is not “which imputer is fanciest?” It is “what process made this value disappear?”

A blank is an event, not a value

Suppose X_j is a feature such as income. Define R_j as its missingness indicator:

• R_j = 1 means observed.
• R_j = 0 means missing.

R_j may predict behavior: a customer who leaves income blank may differ from one who reports $55,000. But it can also proxy for a form version, hospital ward, or access to technology.

An imputer estimates missing X_j from the values that remain. A mean imputer uses the overall center; KNN uses nearby rows; iterative imputation uses relationships among columns; a tree model may route missing rows without creating a replacement. Each method makes a different assumption.

MCAR, MAR, and MNAR

MCAR: missing completely at random

A value is MCAR when its chance of being missing is unrelated to both the value itself and the other data.

For example, a factory temperature sensor loses power because a cable connector randomly shakes loose. The failure is unrelated to whether the machine was hot, cold, old, or new.

If genuine, dropping incomplete rows can preserve unbiased relationships, provided enough rows remain. The cost is lost information and statistical power; mean or median imputation still shrinks variation.

MAR: missing at random, conditional on what you observed

A value is MAR when its missingness can be explained by other observed variables. If R is the missingness pattern and a row is divided into observed and missing components, MAR means:

P(R | X_observed, X_missing) = P(R | X_observed)

Suppose mobile applicants are twice as likely to skip income, while device_type, age, employment status, and region are observed. If those fields fully explain the difference, income is MAR conditional on them.

“Random” here does not mean unrelated to the data. Mobile users may have a different income distribution, so a conditional imputer can use device type and related features while a global mean cannot.

MNAR: missing not at random

MNAR means the MAR equality fails: after accounting for observed variables, missingness still depends on unobserved information, often the missing value itself.

For example, high-income applicants may be less willing to report income. Applicants with the same device, age, and industry can still have different reporting probabilities because of their unobserved income.

Different hidden distributions can produce the same visible dataset. No algorithm can identify the true missing values from observed columns alone without additional assumptions. Use domain knowledge, improve collection, or run sensitivity analyses such as “what if missing incomes are 20 percent higher than the MAR estimate?”

The mechanism can differ by feature and subgroup. income might be MNAR while zip_code is missing because of a broken import.

Start with the least complicated defensible baseline

Inspect the source process before imputing. Count missing values by feature, date, device, site, customer segment, and target class. Check whether an empty value means “not applicable”: a renter may have no landlord contact, which is not a failed measurement.

Do not automatically impute the target. If a label is absent, remove that training example, obtain the label later, or define a different prediction task.

For modest, plausibly MCAR numeric missingness, median imputation is a useful baseline because it resists extreme values. For categorical data, an explicit missing category can be more honest than the mode.

Why mean imputation shrinks variance

Take four rows where x and y are perfectly aligned:

Row	True x	Observed x	y
1	1	1	1
2	2	2	2
3	3	3	3
4	4	missing	4

The observed values of x have mean 2. Mean imputation changes row 4 to 2:

x_imputed = [1, 2, 3, 2]

The true x values have mean 2.5 and population variance 1.25 (dividing squared deviations by four). The imputed values have mean 2 and population variance 0.5.

The imputed column has less than half the original spread. Every replacement equals the observed mean, so it contributes zero deviation from that mean.

It also changes correlation. The true correlation between x and y is 1. After imputation, the deviations of x are [-1, 0, 1, 0], and those of y are [-1.5, -0.5, 0.5, 1.5]. The Pearson correlation is:

2 / sqrt(2 * 5), or about 0.63.

One replacement weakened a perfect relationship because the missing row was placed at the center of x even though its y was high. Mean imputation often pulls correlations toward zero; a median has the same pile-up problem, though it is more robust to outliers.

Add an indicator when absence may carry information

A missingness indicator is a binary feature recording whether the original value was missing. Impute x with a median, then provide both x_imputed and M_x to the model.

A linear model can learn:

prediction = beta_0 + beta_1 * x_imputed + beta_2 * M_x

This distinguishes “income is near the median” from “income was not supplied and the median is only a placeholder.” It preserves the information that the field was absent; it does not reconstruct income.

In scikit-learn, SimpleImputer(strategy="median", add_indicator=True) adds indicators only for features missing during fit. If a feature is complete during fitting but missing in production, it is imputed but receives no indicator. Use MissingIndicator(features="all") in parallel when every feature needs one.

KNN and iterative imputation

K-nearest-neighbors imputation fills a blank using similar rows. It works when similarity in observed fields predicts similarity in the missing field. Scale features with training-derived statistics: otherwise dollars can dominate age. Irrelevant or numerous columns can make neighbors meaningless, and missing coordinates may leave too little information for a useful distance. KNNImputer can be expensive on large datasets.

Iterative imputation models each incomplete feature from the others. For age, income, and credit utilization, it repeatedly predicts each column from the remaining columns until values stabilize. This is the intuition behind MICE (Multiple Imputation by Chained Equations).

Iterative methods can preserve relationships that a global median destroys when features strongly predict one another and MAR is credible. They are model-dependent: poor conditional models can extrapolate to negative balances or impossible ages. MICE properly means stochastic chained imputations that create several completed datasets, not merely one deterministic table. For inference, fit the analysis to each dataset and pool the estimates; a single completed dataset usually understates uncertainty.

A predictive imputer must use only information available at prediction time. Using the historical outcome to fill a missing predictor may be valid in some offline analyses, but it is label leakage in a serving pipeline.

Tree models may not need an imputer

XGBoost and LightGBM handle missing numeric values natively. XGBoost learns a default direction for missing values at each split; LightGBM uses a missing-value bin and learns a route for it.

Native handling can preserve NaN values and let the tree learn that a blank income follows a different branch from a blank utilization. It does not solve MNAR identification, repair an upstream feed, or guarantee fair behavior, and it does not apply to every estimator, such as logistic regression.

Benchmark native handling against a leakage-safe imputed pipeline using identical folds. See XGBoost for split mechanics.

Choosing a method

Deciding situation	Candidate	Why it fits	Main cost or risk
Linear or distance-based model	Median or mean plus indicator	Fast and stable	Shrinks distributions
Missing value has a business meaning	Explicit category or domain rule	Preserves “not applicable”	Rules can change
Local similarity is meaningful	KNN	Uses nearby rows	Sensitive to scale and dimensions
Strong feature relationships; MAR credible	Iterative imputation or MICE	Models conditional relationships	Slower and model-dependent
XGBoost or LightGBM	Native handling	Learns split-specific missing routes	Requires stable patterns and model support
Plausibly MNAR	Sensitivity analysis and process knowledge	Makes assumptions visible	Cannot identify hidden truth alone

Do not judge only imputation error. Mask observed values and test recovery, then evaluate the downstream model. Random masking approximates MCAR; group-based masking explores MAR; systematically higher or lower hidden values explore MNAR. These tests show sensitivity, not what happened in production.

The leakage rule: split first, fit second

An imputer learns a median, neighborhoods, or conditional models. It must not learn from validation or test rows.

Inside each cross-validation fold:

1. Split into training and validation portions.
2. Fit the imputer on training data only.
3. Transform both portions with that fitted imputer.
4. Fit the model on transformed training data.
5. Score the untouched validation portion.

In scikit-learn, put the imputer and estimator in one Pipeline:

import numpy as np

from sklearn.datasets import make_classification
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(
    n_samples=240, n_features=5, n_informative=4,
    n_redundant=0, random_state=7,
)

rng = np.random.default_rng(7)
X[rng.random(X.shape) < 0.12] = np.nan

model = Pipeline(
    steps=[
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
        ("predict", LogisticRegression(max_iter=1000)),
    ]
)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc")

print(f"fold AUCs: {np.round(scores, 3)}")
print(f"mean AUC: {scores.mean():.3f}")

Each fold learns its median from training rows only. This random stratified split suits synthetic iid data, not necessarily production. Use time-based splits when missingness changes over time, and GroupKFold or StratifiedGroupKFold when customers, sites, or related records must stay together. Keep the final future- or group-held-out test set untouched.

The dangerous version is imputer.fit_transform(X) before cross-validation. Even without labels, the imputer has learned from supposed holdout rows. The same rule applies to scaling, feature selection, rare-category grouping, outlier thresholds, and target encoding. See data leakage and train, test, and cross-validation.

After selecting the model, fit the complete pipeline on all available training data, but not the final test set until final evaluation.

Failure modes you can catch early

First symptom	Likely cause	Fix
CV AUC is 0.91, but production is 0.74	Leakage or changed missingness process	Put transforms in the pipeline; monitor missingness
Coefficients are small and many rows share one value	Mean or median created a pile	Add an indicator; compare conditional methods
Serving raises Input X contains NaN	Inference does not use the fitted imputer	Serialize and serve the same pipeline
Indicator dominates for one group	Proxy for access or collection policy	Audit group rates, fairness, and the collection process

Alert on raw missingness patterns. If a form release raises missingness from 8% to 35%, no imputer can make it an ordinary day.

What imputation cannot do

Imputation creates a plausible value, not an observed fact. If income is MNAR, a smooth model-based distribution can still be wrong where it matters. If a safety-critical feature is half missing, collecting it, deferring the decision, or exposing uncertainty may be better than making a confident guess.

Native tree handling, indicators, KNN, and MICE address different assumptions; none replaces understanding the source system. Compare with a simple baseline, validate inside the fold, inspect subgroup rates, and stress-test plausible MNAR scenarios.

Quick check

0/3

Q1

MCAR, because the blank is not caused by the income valueMAR, conditional on the observed device typeMNAR, because every skipped value is MNARThe mechanism cannot affect the imputer

Check answer

Q2

It rises from 0.5 to 1.25It stays at 1.25It falls from the true 1.25 to 0.5It becomes zero because one value was missing

Check answer

Q3

Assume MCAR and drop all rows with missing blood pressureUse a global mean and treat the problem as solvedTreat MAR as a possible partial explanation, investigate MNAR, and run sensitivity analysesFit the imputer on the full dataset so it sees more blood-pressure patterns

Check answer