The 3 a.m. ML page — incident response

The last lesson gave us the calm response to a model going wrong: a scheduled retrain, a measured trigger, a challenger evaluated at leisure. And then it admitted the obvious — that calm is a luxury you don’t always get. Sometimes the model isn’t drifting; it’s failing right now, with users watching and the building on fire. We asked what you do in the first ten minutes. Here is the playbook.

It’s 3:14 a.m. on a Saturday. PagerDuty buzzes. The dashboard your team set up six months ago says churn_model_recall_24h = 0.21, down from 0.78 yesterday. Customer support has three escalations from premium accounts saying they got the wrong onboarding flow. Slack has a thread in #oncall that started two hours ago.

What you do in the next ten minutes determines whether this is a story you tell at standup or one you tell at the next quarterly review.

Step 1 — rollback first, debug later

The single most important rule of ML on-call:

Don’t try to fix it. Roll it back.

Production is the wrong place to debug. Every minute the broken model is serving traffic, you’re accumulating bad predictions that may also be contaminating the next training set (feedback loop), generating user complaints, and burning the trust people had in the service.

Restore the last-known-good model. Then investigate what went wrong with what you just unshipped.

If you set up the MLflow Model Registry properly, rollback is one command. If your serving code loads models:/churn/Production by name and you transitioned the previous version back to Production, the next service restart picks up the old model. Done. Buy yourself hours of debugging time without the customer-impact clock running.

# Pseudocode for the only commands you should run in the first 10 minutes.
mlflow.transition_model_version_stage(
    name="churn-classifier",
    version=PREVIOUS_GOOD_VERSION,
    stage="Production",
)

# Restart serving pods so they reload the new "Production" version.
kubectl rollout restart deployment/churn-service

A rollback that takes ten minutes is hugely better than a “real fix” that takes three hours. Fix-forward means diagnosing, writing, testing, and deploying new code under pressure at 3 a.m. — each step an opportunity for a second mistake. Rollback is a single known-good artifact. The risk profile is incomparable.

Step 2 — once rolled back, debug methodically

You’ve bought time. Now figure out what broke. ML incidents almost always come down to one of three categories. Triage in this order:

The order matters. Data issues are the most common in mature ML systems, but they’re also the easiest to mistake for “model is bad.” Look at the inputs first.

Concrete checks for each:

# DATA — five-minute diagnostic
df = load_recent_inputs(window="6h")
print(df.isna().sum())              # any feature suddenly all NaN?
print(df.describe())                # means/stds vs training time?
print(df.dtypes)                    # did any column change type?
print((df == 0).mean())             # any feature suddenly all-zero?

# CODE — what changed?
git log --since="48 hours ago" --oneline app/ training/

# INFRA — what's the serving layer saying?
kubectl logs -l app=churn-service --tail=200 | grep -E "(ERROR|OOM|model)"

Most incidents are obvious within ten minutes of the right diagnostic. The hard ones aren’t hard because the cause is exotic — they’re hard because nobody thought to look at the inputs.

Step 3 — canary and shadow, so this happens less

Once you’re past this incident, harden the deploy path. Two patterns, both standard:

Canary deploy

A new model version gets a small fraction of traffic (5–10%) for a short window. During that window, you compare its predictions and its downstream business metrics to the old version. If anything regresses, abort. If it looks fine, ramp up.

Canary is the difference between “5% of users saw a broken model for two hours” and “100% of users saw a broken model for two hours.” Both hurt; the first one hurts 20x less.

Shadow deploy

The new model receives every request but its predictions are discarded (or logged for offline comparison). It serves no traffic. You compare its outputs to the current production model on real, live data without exposing users.

Shadow is the right pattern when:

• You can’t tolerate any user-facing risk (regulated industries).
• You want to validate behaviour on the real traffic distribution before any rollout.
• You need ground truth to arrive before you can really evaluate (canary’s “downstream metric” can’t always be measured in 10 minutes).

Most teams shadow for 1–7 days, then canary for a few hours, then full rollout. It’s slower and it’s how you stop being on-call at 3 a.m.

Step 4 — the ML-specific postmortem

A postmortem (also called a “retrospective” or “incident review”) is a written record produced after an incident is resolved: what happened, why, what the impact was, and what action items prevent recurrence. The “blameless” variant focuses on system failures, not individual mistakes.

The blameless postmortem template you use for software is almost right. ML adds one column. Use this:

# Postmortem — Churn model recall regression, 2026-05-25

## Summary
At 03:14 UTC, recall on the production churn model dropped from 0.78 to
0.21 over a 6-hour window. Mitigated at 03:38 UTC by rolling back to
version 14. Root cause: upstream feature pipeline change merged ~26h
prior, which renamed `last_payment_amount_usd` to
`last_payment_amount_cents` without coordinating with the model
service. The serving feature pipeline silently produced zeros for the
unrecognised column.

## Impact
- ~2h 24m of degraded predictions
- 14 customer complaints
- ~$X estimated revenue impact (lost reactivation campaigns)

## Timeline
- 01:14 UTC: feature pipeline deploy of `data-platform v2.18`
- 01:30 UTC: serving started producing 0.0 for `last_payment_amount_usd`
- 03:14 UTC: monitoring alert (`recall_24h < 0.5` threshold)
- 03:18 UTC: on-call paged
- 03:25 UTC: confirmed model behaviour was off
- 03:29 UTC: rolled back to model v14 (which used a different feature)
- 03:38 UTC: traffic stabilised, recall returned to 0.74

## Was it data, code, or infra?
DATA. Upstream feature schema changed; serving feature pipeline did
not detect the rename and produced default values.

## Why didn't we catch it?
- The feature pipeline did not have a schema-validation step on output.
- The serving service did not validate that all expected features were
  present and non-zero before predicting.
- Our drift dashboard was sampled hourly, not real-time, and the
  alert threshold for that feature was too lax.

## Action items
- [ ] (HIGH) Add a presence + range check on every required feature in
      the serving pipeline; reject the request with an explicit error
      if a feature is missing.
- [ ] (HIGH) Add schema-contract test between feature pipeline outputs
      and model service inputs in CI (catches the rename pre-merge).
- [ ] (MED) Tighten drift alert thresholds on top-5 features by
      importance.
- [ ] (MED) Add a "feature distribution dashboard" linked from the
      incident channel.

The “Was it data, code, or infra?” section is the ML-specific bit. It forces you to commit to a category, which makes the action items sharper. Vague “we should monitor more” becomes specific “add a schema contract test in CI.”

A worked example — the cents-to-dollars story

The postmortem above is fictional but the failure mode is real and common. Walk through what you’d actually have done as on-call:

1. 03:18 — page received. Open dashboard. Confirm recall_24h has dropped. Confirm prediction distribution: today’s mean predicted churn risk is 0.04, vs historical mean of 0.30. Big tell that something’s off in the inputs, not the model.
2. 03:25 — pull recent input rows from the logging table. Notice last_payment_amount_usd is 0.0 for 100% of recent rows. Was it always? No — last week it was averaging ~120.
3. 03:27 — git log the feature pipeline. Last deploy was 26h ago. Commit message: “rename amount columns to specify unit.” Now you know.
4. 03:29 — roll back the model version. Model v14 was trained before the rename and uses a different feature; restoring it removes the dependency on the missing column.
5. 03:38 — metrics stabilise. Incident channel: “Mitigated. Root cause: feature rename upstream. Postmortem to follow Monday.”
6. Monday — write the postmortem. File the action items. Add the schema-contract test that would have caught this in CI.

You didn’t fix the bug at 3 a.m. You bought time and let the daytime team fix it properly with sleep, coffee, and code review. That’s the job.

A small “rollback choreography” script

You can’t run K8s in the browser, but you can sketch the logic of “resolve the previous Production version and transition it back” — which is the script you should have ready before you need it.

# Simulated rollback choreography — what your runbook does.

class FakeRegistry:
    """Stand-in for MLflow's MlflowClient."""
    def __init__(self):
        self.versions = [
            {"version": 12, "stage": "Archived"},
            {"version": 13, "stage": "Archived"},
            {"version": 14, "stage": "Archived"},
            {"version": 15, "stage": "Production"},   # the broken one
        ]

    def search_model_versions(self, name):
        return [v.copy() for v in self.versions]

    def transition(self, name, version, stage):
        for v in self.versions:
            if v["version"] == version:
                v["stage"] = stage
            elif v["stage"] == stage:
                v["stage"] = "Archived"

def rollback(client, model_name="churn-classifier"):
    versions = client.search_model_versions(model_name)
    current = next(v for v in versions if v["stage"] == "Production")   # the broken one
    # most recent Archived version BEFORE the current Production version
    candidates = [v for v in versions if v["stage"] == "Archived" and v["version"] < current["version"]]
    if not candidates:
        raise RuntimeError("no rollback candidate available — the failure mode that ruins your night")
    target = max(candidates, key=lambda v: v["version"])
    client.transition(model_name, target["version"], "Production")
    return current["version"], target["version"]

reg = FakeRegistry()
old, new = rollback(reg)
print(f"rolled back: v{old} -> v{new}")
print("registry state:", reg.search_model_versions("churn-classifier"))

rolled back: v15 -> v14
registry state: [{'version': 12, 'stage': 'Archived'}, {'version': 13, 'stage': 'Archived'}, {'version': 14, 'stage': 'Production'}, {'version': 15, 'stage': 'Archived'}]

Read the registry state after the call: the broken v15 has been demoted to Archived, and v14 — the most recent good version before it — now holds Production. That is the entire rollback: one function, no new code, no decisions. The single line that earns its keep is the RuntimeError — if no archived candidate exists (because someone deleted old versions to save storage), the script fails loudly here, in calm daylight when you’re writing the runbook, instead of at 3 a.m. when you discover you have nothing to roll back to.

The point isn’t the code — it’s that you have a named runbook step, “rollback”, that does exactly one thing and you can execute it in the first ten minutes without thinking. Pre-built. Tested. Documented in the runbook.

In one breath

When a model is failing in production, the only correct first move is roll back, don’t debug — restore the last-known-good version (one command if your registry resolves models by name and preserves old versions, which is the single habit that makes rollback possible at all), then triage methodically through data → code → infra in that order (data issues are the most common and the easiest to mistake for “the model is bad”); harden the deploy path afterward with canary and shadow so the next failure shows up small or invisible, and close every incident with a blameless postmortem whose ML-specific question — was it data, code, or infra? — forces vague regret into a concrete action item.

Practice

Before the quiz, internalize the ordering. It’s 3 a.m., recall has cratered, and your instinct is to open the model code and start reading. Argue why that instinct is exactly wrong — what is the first command you run, and what clock does it stop? Then the triage: the lesson insists you check data before code before infra. In the cents-to-dollars worked example, which category was it, and what single five-minute diagnostic would have pointed straight at it?

A question to carry forward

Read back over the playbook and notice the quiet assumption holding it all together: that you can roll back. The whole strategy rests on a versioned, deterministic artifact — model v15 was broken, so swap in the v14 that sat in your registry, and the world is restored. Every rollback in this lesson is a clean swap of a file you own and control.

Now imagine the model isn’t a .joblib in your registry. It’s a large language model behind an API — maybe one a vendor updated underneath you last night, with no version for you to pin; maybe one whose failure isn’t a recall drop but a confident hallucination, or a prompt-injection that turned your support bot hostile, or a token bill that 10בd overnight. “Roll back to the previous artifact” assumes an artifact and a deterministic output, and an LLM gives you neither cleanly. So the question to carry forward, into the last lesson of this chapter, is: when the thing in production is an LLM, how does everything we’ve built — versioning, evaluation, monitoring, rollback, cost control — have to change? That is LLMOps, and it is next.