The real ML lifecycle

We closed the recommenders section on a question that turns out to belong to every model, not just the ones that suggest movies. You can build the thing — train it, tune it, watch it beat your benchmark — but how do you take a model that works on your laptop and turn it into a system that keeps working in production, served and monitored and versioned and retrained, for as long as anyone depends on it? That question is the whole of this section, and the cleanest way into it is a story.

Last Tuesday, a checkout-prediction model that had served happily for ten months started returning nonsense. The on-call engineer rolled it back in five minutes. The real fix took two days: a payment-provider upgrade had switched a total column from cents to dollars. Training data: cents. Live traffic: dollars. The model was technically working — it just lived in a world that no longer existed.

That is the defining fact of production machine learning. MLOps (Machine Learning Operations — the practice of keeping models alive and honest in production) is the discipline of treating that story as the default, not the exception. The model that beat your offline benchmark is the start of the job, not the end.

The loop, not the pipeline

If you draw the actual lifecycle of a production model, it isn’t a line. It’s a loop, and you keep going around it for as long as the model is serving traffic.

This is the same lesson software engineering learned decades ago — ML just takes it one step further. Classic waterfall development runs once through a fixed sequence — requirements, design, build, test, release — and treats “done” as a real, final state. Agile rejected that: instead of one long pass, you loop through small build → measure → learn iterations (“sprints”), shipping a little and adjusting from feedback. ML keeps agile’s loop but removes the exit — because even a frozen, “finished” model quietly decays as the world it learned from drifts away. In ML, there is no “done.”

The loop below is what that never-ending cycle looks like for a model in production.

Every arrow is a place where things go wrong, and every node is something you’ll eventually have to automate. That’s MLOps.

Stage by stage — and how each one bites

1. Data

You don’t really “have a dataset.” You have a snapshot of a database that will look different tomorrow. The dataset you trained on is already stale the moment training finishes.

Bites you with: schema drift (a column you depend on changes type), silent backfills (someone re-runs ETL and your training labels shift), leakage (a feature derived from the target sneaks into your input).

We’ll cover this in baselines (catching leakage early) and data-centric (fixing labels and slices, not models).

2. Train

The interesting bit, and the part that gets too much attention. If your training script isn’t deterministic and reproducible, the model that shipped is a one-off artifact nobody can rebuild.

Bites you with: missing seeds, library version drift, “it works on my laptop” Python environments, weights you can’t trace back to a commit.

This is the MLflow lesson — pin every experiment to a run ID, log every artifact, and pin every dependency.

3. Eval

Where the false sense of safety lives. A single test-set number tells you almost nothing about whether the model is better than what’s running now, on the slices that actually matter.

Bites you with: one metric on the whole dataset (the model is great on average and terrible for your top 5 customers), no comparison to a baseline (the 0.91 F1 looks great until you realise a constant predictor gets 0.90), no slice-level breakdown.

4. Deploy

Code makes it to a server. Now you find out whether the model you eval’d is the model you shipped. Spoiler: usually it isn’t, because your training preprocessing and your serving preprocessing diverged.

Bites you with: training/serving skew (when the feature preprocessing at training time and at inference time diverge — same column name, different computation), latency surprises, version mismatches between the model file and the runtime that loads it.

Covered in Docker for ML, FastAPI serving, and BentoML.

5. Monitor

The model is live. Now what? Most teams’ answer is “we’ll find out when a customer complains.” That’s the most expensive monitoring strategy ever invented.

Bites you with: drift (the world changed, the model didn’t), silent degradation (accuracy is dropping but accuracy isn’t logged), feedback loops (your model’s outputs become the next month’s training data).

Covered in drift and incident-response.

6. Retrain

When monitoring says “this is bad,” you retrain. But retrain on what? On the data that contains the drift you’re trying to catch up to? Yes — carefully, with the same eval gates as the first model.

Bites you with: retraining on biased post-deployment data, no rollback path when the new model is worse than the old one, no canary/shadow to catch regression before traffic hits.

Covered in GitHub Actions (the retrain trigger) and incident-response (the rollback).

The “ops” part — and why it’s a loop

Look at the diagram again. The arrow from Monitor back to Data is where MLOps actually earns its name. Without that arrow, you have ML research. With it, you have a system that adapts.

Concretely, the loop means three things you’ll set up:

1. Triggered retraining. When a drift threshold trips or a scheduled window comes up, a CI pipeline pulls fresh data, retrains, and runs the eval suite — all without you opening a notebook.
2. A model registry. Every trained candidate has a version, lineage (which commit, which data snapshot), and a stage (Staging, Production, Archived). You can roll back in one command.
3. A monitoring loop. Predictions, ground truth (when it arrives), and feature distributions all flow into observability. You don’t ship a model — you ship a model plus the loop that catches it when it falls.

A quick reality-check exercise

This lesson is mostly conceptual, but let’s at least make the loop concrete in code — even a toy version. Imagine the simplest possible “model”: predict the mean of y. The point isn’t the model; it’s the loop.

import numpy as np

# A toy world that drifts over time: the true mean of y creeps upward with t.
def next_batch(t, n=200, rng=np.random.default_rng(0)):
    return rng.normal(loc=10 + 0.5 * t, scale=2.0, size=n)

# Our "model" is just the running mean from a frozen training batch.
training_y = next_batch(t=0)
model_prediction = training_y.mean()
print(f"trained model predicts:  {model_prediction:.2f}")

# Simulate weekly production batches and "monitor" the drift.
threshold = 1.5      # retrain when error exceeds this
for week in range(1, 6):
    batch = next_batch(t=week)
    actual_mean = batch.mean()
    error = abs(actual_mean - model_prediction)
    flag = "RETRAIN" if error > threshold else "ok"
    print(f"week {week}: actual={actual_mean:.2f}  predicted={model_prediction:.2f}  err={error:.2f}  [{flag}]")
    if error > threshold:
        model_prediction = batch.mean()   # retrain on the latest batch

trained model predicts:  10.03
week 1: actual=10.32  predicted=10.03  err=0.29  [ok]
week 2: actual=11.01  predicted=10.03  err=0.98  [ok]
week 3: actual=11.43  predicted=10.03  err=1.40  [ok]
week 4: actual=11.73  predicted=10.03  err=1.70  [RETRAIN]
week 5: actual=12.60  predicted=11.73  err=0.87  [ok]

Watch the drift do its quiet work. The model froze at 10.03, and for three weeks the world stays close enough — the error climbs 0.29, 0.98, 1.40, never quite tripping the 1.5 threshold. Then week 4 crosses it at 1.70, the monitor fires RETRAIN, the model snaps to the fresh mean of 11.73, and week 5’s error collapses back to 0.87. That is the whole lifecycle in twenty lines: train, predict, monitor, detect drift, retrain, repeat. Production versions are messier — but the shape is exactly this.

In one breath

The real machine-learning lifecycle is not a train-and-ship line but a closed loop — data, train, eval, deploy, monitor, retrain — where the monitor-back-to-data arrow is the part that earns the word “ops”; because a frozen model quietly decays as the world it learned from drifts, there is no “done,” so the senior move is to build the loop (registry, eval gates, monitoring, rollback) first and slot the model into a hole that is already waiting for it.

Practice

Before the quiz, reason about the drift simulation. Week 4 tripped RETRAIN at error 1.70, and retraining dropped week 5 back to 0.87. If you lowered the threshold from 1.5 to 0.9, which weeks would retrain instead — and what is the cost of a monitor that is too twitchy? Then map the opening cents-to-dollars story onto the six stages: which stage should have caught it before it ever reached production, and which stage actually did?

A question to carry forward

You have now seen the loop and the six places it bites. But notice the very first node — Eval — quietly assumed something we never justified: that you can tell whether a model is any good. The lesson flagged the trap in passing (“the 0.91 F1 looks great until you realise a constant predictor gets 0.90”), then moved on. It deserves better, because it is the question that decides whether your whole loop is measuring progress or fooling you.

So the question to carry forward is the most deflating one in machine learning: before you are allowed to call a model “good,” what exactly does it have to beat? Not another model — something far dumber, and far more revealing. The next lesson builds that floor: the baseline, the cheapest model you will ever write and the one that most often saves you from shipping something worse than a constant.