Drift — the silent killer of deployed models

The last lesson left us with a model that is, at last, correct — skew killed, train and serve producing identical numbers, genuinely right on launch day. And we ended on the unsettling thing about that victory: it is exactly when the slow death begins. A skew-free model is born correct and then watches the world walk away from it, one quiet week at a time. This lesson is how you notice.

A deployed model doesn’t fail loudly. Accuracy doesn’t drop to zero overnight. What actually happens: a feature distribution slowly shifts, the model’s predictions get progressively worse, and three months later someone notices revenue is down 4%. That’s drift — when the real world no longer matches the world the model was trained on, so its learned patterns no longer apply. It’s the single most expensive failure mode in production ML.

The good news: drift is measurable. The bad news: it has two flavours that look identical from a metrics dashboard but require different fixes.

Two flavours of drift

The crucial subtlety: data drift is easy to detect, concept drift is hard. Detecting data drift requires only the inputs the model sees. Detecting concept drift requires ground truth labels, which typically arrive late, partially, or never. Most production “drift detection” is really data-drift detection plus prediction-distribution monitoring as a proxy for the harder problem.

A useful mental model:

Detector #1 — KS test on continuous features

Kolmogorov-Smirnov. Compares two empirical distributions. Returns a statistic + a p-value telling you how unlikely it is that the two samples came from the same distribution. Cheap, no parameters, works out of the box.

import numpy as np
from scipy.stats import ks_2samp

rng = np.random.default_rng(0)

train = rng.normal(loc=50, scale=10, size=2000)   # training distribution
week1 = rng.normal(loc=50, scale=10, size=500)    # week 1 — same world, sampling noise
week4 = rng.normal(loc=56, scale=10, size=500)    # week 4 — mean has shifted up by 6

r1 = ks_2samp(train, week1)
r4 = ks_2samp(train, week4)
print(f"week 1 vs train: statistic={r1.statistic:.3f}  p={r1.pvalue:.3g}")
print(f"week 4 vs train: statistic={r4.statistic:.3f}  p={r4.pvalue:.3g}")

week 1 vs train: statistic=0.038  p=0.602
week 4 vs train: statistic=0.238  p=1.96e-20

The two weeks tell opposite stories. Week 1 is the same distribution sampled afresh, and the test knows it: a tiny statistic of 0.038 and a p-value of 0.60 — a 60% chance of seeing this much wiggle from pure sampling noise, so nothing to act on. Week 4, where the mean crept from 50 to 56, lights up: the statistic jumps to 0.238 and the p-value collapses to 2e-20, meaning a shift this large essentially never happens by chance. Read the two pieces:

• statistic near 0 means the two samples look identical; near 1 means they’re very different.
• p-value is the probability of seeing this much difference if the underlying distributions were actually the same. Small p-value (e.g. < 0.01) → reject “no drift.”

In production you run KS per feature, per day, and alert when statistic > threshold (a thresholded statistic is more useful than a p-value alone — p-values get tiny on large samples even for trivial shifts).

Detector #2 — PSI (Population Stability Index)

PSI is the risk-modeling world’s drift detector. It bins both distributions into the same buckets, then computes a weighted log-ratio of bin frequencies. Rules of thumb that have stuck for decades:

import numpy as np

def psi(expected, actual, bins=10):
    """Population Stability Index — bin both, compute weighted log-ratio."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))   # bins from expected
    edges[0], edges[-1] = -np.inf, np.inf                        # capture tails
    e_pct, _ = np.histogram(expected, bins=edges)
    a_pct, _ = np.histogram(actual, bins=edges)
    e_pct = np.clip(e_pct / e_pct.sum(), 1e-6, None)
    a_pct = np.clip(a_pct / a_pct.sum(), 1e-6, None)
    return float(np.sum((a_pct - e_pct) * np.log(a_pct / e_pct)))

rng = np.random.default_rng(0)
train = rng.normal(50, 10, 5000)
week1 = rng.normal(50, 10, 1000)        # same distribution
week4 = rng.normal(56, 10, 1000)        # mean shifted up
week8 = rng.normal(50, 16, 1000)        # variance widened

print(f"PSI train vs week1: {psi(train, week1):.4f}  (expect ~0)")
print(f"PSI train vs week4: {psi(train, week4):.4f}  (expect > 0.1)")
print(f"PSI train vs week8: {psi(train, week8):.4f}  (expect > 0.1)")

PSI train vs week1: 0.0096  (expect ~0)
PSI train vs week4: 0.3583  (expect > 0.1)
PSI train vs week8: 0.3166  (expect > 0.1)

PSI tells the same story as KS, in the dialect risk teams speak. Week 1 scores 0.0096 — comfortably under the 0.1 “no shift” line. Week 4’s mean-shift lands at 0.3583, well past the 0.25 “major shift, act” threshold. And week 8 is the instructive one: the mean didn’t move at all, only the variance widened (scale 10 → 16) — yet PSI still catches it at 0.3166, because spreading the distribution moves mass between bins just as surely as shifting it does. A detector that only watched the average would have missed week 8 entirely.

KS vs PSI — practical guidance:

• KS is great for continuous features with sample sizes you can control. Cheap, no binning decisions.
• PSI is the lingua franca for credit-risk and insurance teams and is easier to communicate to non-technical stakeholders (“PSI > 0.25, investigate” is a clearer alert than “p < 0.01 with statistic 0.12”).
• Use both. They agree on big shifts, and disagreement is itself signal.

For categorical features, swap PSI’s histogram for category counts — the formula stays the same. KS doesn’t apply directly to categoricals; use chi-squared instead.

The cheapest detector — prediction-distribution monitoring

Forget the inputs for a second. Just log every prediction your model makes. Plot the distribution of those predictions over time. If your churn classifier used to say “30% churn risk on average” and is now saying “55% on average,” something changed — input drift, concept drift, an upstream bug, or all three. You won’t know what without deeper analysis, but you’ll have caught the symptom in approximately zero engineering effort.

# Pseudocode for the cheapest drift monitor you'll ever write.
# Run nightly on yesterday's predictions.
mean_pred = predictions["proba"].mean()
historical_mean = 0.30
delta = abs(mean_pred - historical_mean)
if delta > 0.10:
    alert(f"Mean predicted churn moved by {delta:.2f} — investigate")

This catches more real problems than any KS test, more quickly, with no infrastructure beyond the prediction log. Build this first, then add the feature-level detectors.

What the tools give you

You can hand-roll KS and PSI as shown above — it’s ~100 lines. The tools earn their keep by handling the boring parts at scale.

NannyML deserves a special call-out: estimating model performance without labels is the actual hard problem in production drift monitoring. Don’t ignore it.

When ‘drift detected’ should — and shouldn’t — trigger a retrain

This is the question that separates senior MLEs from juniors.

Don’t auto-retrain on every drift alert. Here’s why:

• Drift can be a signal of an upstream bug. A field changed type from cents to dollars; your model now sees inputs 100x larger; KS screams. Retraining on broken data ships a model that learned the bug.
• Drift can be seasonal. Holiday traffic looks different from January traffic. Retraining on Black Friday data and deploying on January 2nd is worse than the original model.
• Drift can be concept drift in a way that requires new labels, not retraining on stale ones. If the relationship between input and output changed, retraining on yesterday’s data doesn’t help — you need fresh ground truth that reflects the new world.

The right pipeline is:

1. Drift detected → open an incident, don’t deploy.
2. Investigate: data quality? schema? upstream change? seasonality?
3. If the drift is real and the labels exist: retrain on the most recent window, evaluate against the current production model, and only promote if it wins.
4. If the labels don’t exist yet, deploy targeted label collection first.

In one breath

Drift is the slow decay of a once-correct model as the world moves away from its training data, in two flavours that look identical on a dashboard but differ in cost: data drift (P(X) shifted — detectable from inputs alone, cheap) and concept drift (P(y|X) shifted — needs ground-truth labels that arrive late or never); detect it with KS tests and PSI on features (PSI even catches a variance-only shift the mean would hide), but build the cheapest detector first — log every prediction and alert when the daily mean moves — and treat any alert as a smoke alarm: investigate before you retrain, because the “drift” might be an upstream bug or a seasonal swing, not a reason to ship a model trained on broken data.

Practice

Before the quiz, look again at week 8 in the PSI run: the mean never moved, yet PSI flagged 0.32. Explain why a naive “alert if the average shifts” monitor would have slept through a real distribution change — and what that says about watching only one summary statistic. Then the seniority question the lesson hammers: a KS test on customer_age jumps to 0.32 overnight. Name two innocent explanations that would make auto-retraining the worst possible response.

A question to carry forward

This lesson kept stopping short on purpose. Over and over — the seniority callout, the “should I retrain?” section, the smoke-alarm framing — it told you what not to do when drift fires, and then deferred the actual cure. “Open an incident, investigate, and if the drift is real and the labels exist, retrain on the recent window, evaluate against production, promote only if it wins.” Notice how much unexamined machinery is hiding inside that one sentence.

Because retraining is not a button. Which window of data — and how do you avoid training on the very upstream-bug or seasonal blip that set off the alarm? Retrain from scratch every time, or update incrementally? How often is too often, and when does a fresh model quietly become worse than the one it replaces? Drift detection is the smoke alarm; it tells you something is burning but not how to put it out without flooding the house. So the question to carry forward is the one the alarm hands off to: once you’ve confirmed the drift is real, how do you actually retrain — safely, on the right data, at the right cadence — without shipping a regression? That is retraining and continual learning, and it is the next lesson.