Retraining & continual learning

At 09:00 on Monday, a recommendation system looks healthy. By Friday, its homepage click-through rate has fallen from 8.1 percent to 6.4 percent. The catalog changed. A new customer segment arrived. One upstream feature is now missing for 12 percent of requests.

Someone says, “Retrain it.”

That sentence sounds like a button. It is not. Which data should the model see? Are the new labels trustworthy yet? Did the model cause the apparent change by choosing different items? How do you know the replacement is better rather than merely different?

A careless answer ships a regression every Friday. An overly cautious answer leaves a stale model serving for six months.

Drift detection tells you that the world, or the data describing it, has changed. Retraining fits a new model using newer or better-selected data. It closes the loop, but does not guarantee improvement. The useful question is when to retrain, with what data, and behind which safety gates.

Scheduled versus trigger-based retraining

A scheduled retrain runs on a fixed cadence: nightly, weekly, or monthly. It is predictable and easy to budget, but the calendar does not know when the business changes. It may rebuild the same model while nothing changes or react too slowly to a product launch.

A trigger-based retrain starts when a measured signal crosses a rule. Signals can include:

• a feature distribution moving far from training data;
• performance falling below a service-level objective, or SLO;
• enough new, validated labels becoming available;
• a business change such as a new country, category, or pricing policy.

A drift alert says inputs changed, not that a new model will be better. A performance alert is stronger, but labels may be delayed or broken. A volume trigger says there is more data, not that the data is useful.

For the recommender, suppose the latest seven days contain 2 million homepage impressions and 128,000 clicks: a 6.4 percent click-through rate. The previous comparable week had 162,000 clicks from 2 million impressions, or 8.1 percent. A rule might request retraining after the rate stays below 7.2 percent for two consecutive days, provided there are at least 100,000 eligible impressions and the event pipeline passes quality checks. Those checks prevent a small sample or tracking outage from becoming a model update.

A scheduled floor plus triggers provides predictable coverage and responsiveness. Choose the maximum model age and cadence based on how quickly the environment changes, label maturity, evidence volume, training and review cost, risk, and rollback capacity. A fast schedule cannot create evidence that has not arrived. Weekly retraining might suit this recommender if behavior changes over days, labels mature within a day, and millions of valid impressions arrive. It is not a universal rule.

Add a cooldown, which blocks duplicate runs for a period, and often hysteresis, which uses different thresholds to start and stop an action. For example, begin retraining when performance stays below 7.2 percent, but declare recovery only above 7.6 percent. Otherwise noise can alternate between “healthy” and “retrain.”

The training window is a product decision

“Train on recent data” hides a consequential choice: what counts as recent?

A seven-day window adapts quickly to a viral product but may overreact to a promotion. A twelve-month window captures seasonality but can drown current behavior in old catalog items and preferences. Compare windows empirically:

These counts are descriptive, not targets. The model learns relationships between user, item, context, and outcome. The table shows why a short window is noisier and a long window slower to adapt.

Use a time-based split: train on earlier events and test on later ones. Randomly scattering events across both sets can leak future behavior into evaluation. The holdout should resemble traffic the candidate will meet after promotion.

Labels require a maturity rule. A click may be immediate; a subscription renewal may take 30 days. If “not observed yet” is treated as “did not happen,” the training set gains false negatives. Define a label cutoff in the dataset query. For a 30-day outcome with a two-day pipeline delay, exclude an event until 32 days after its event time. With an end-of-day UTC convention, a run on August 28 uses events through roughly July 27, not the latest 30 calendar days.

If immature examples must be retained, use a justified method such as survival analysis or inverse-probability-of-censoring weighting (IPCW), which accounts for right-censored outcomes. An ordinary classifier does not become censoring-aware because it receives an “incomplete” flag.

Features must also be point-in-time correct: they may use only information available when the prediction was made. “Purchases in the next seven days” is future leakage, not a production feature.

Quarantine rows from incidents such as a broken feature pipeline before training. Record the dataset snapshot, time range, label definition, feature-code version, and excluded partitions. This provenance explains why a model changed and makes a rerun possible.

What the pipeline does

A safe run is a sequence of gates:

1. Take an immutable snapshot of eligible data so a rerun sees the same inputs.
2. Validate row counts, null rates, label balance, feature ranges, freshness, and incident partitions.
3. Build features with the same point-in-time rules used in serving.
4. Train candidates, evaluate them on a temporal holdout, and record model version, data snapshot, code revision, environment, configuration, and metrics.

A model file without those companions is a mystery with a filename.

Champion versus challenger

The live model is the champion. A newly trained candidate is the challenger. The challenger first needs an absolute quality floor and must not materially regress on important guardrails.

Suppose the recommender’s champion produces these results on a fixed future-dated holdout:

• NDCG@10: 0.412;
• Recall@10: 0.238;
• catalog coverage: 42 percent;
• p95 scoring latency: 86 milliseconds.

NDCG gives more credit when relevant items appear near the top ten positions. Recall@10 is the share of known relevant items retrieved in the top ten.

The challenger records 0.421 NDCG@10, 0.244 Recall@10, 41 percent coverage, and 91 milliseconds. It has better offline ranking scores, but lower coverage and higher latency. If the contract requires p95 latency below 100 milliseconds and coverage above 35 percent, it may pass. If the product is trying to expose more of the catalog, the coverage loss may fail it.

Static logged recommendations cannot directly measure the challenger’s real CTR: it did not control which items users saw. That is the counterfactual problem. Offline gains are useful, but actual CTR or purchase-rate comparisons require randomized online traffic.

Write the promotion rule before viewing results. For example, if the primary online metric is CTR and the required improvement is 0.2 percentage points, promote only when the lower confidence bound for the challenger-minus-champion difference is at least +0.2 points. A non-inferiority rule might instead require the lower bound to remain above −0.2 points. These are different decisions. Predeclare the minimum sample, horizon, eligibility rules, stopping plan, and guardrail uncertainty rules as well.

A shadow deployment copies requests to the challenger without showing its results. It tests feature availability, serialization, memory, and latency, but not user preference. A canary gives a small fraction of real traffic to the challenger. An A/B test assigns users to champion or challenger and compares outcomes under a planned design. A canary is an operational safety step; an A/B test is a measurement design.

In the Monday scenario, give each model 100,000 comparable impressions. The champion receives 8,000 clicks, or 8.0 percent. The challenger receives 8,400, or 8.4 percent: an observed difference of 0.4 percentage points. That is encouraging, not proof. Use a valid analysis, confidence intervals or a statistical test, an adequate sample, and assignment checks. Also inspect purchases, complaints, diversity, latency, and failures. A model can win clicks by recommending cheap clickbait.

Promotion should be boring: change one model reference, keep the previous champion available, watch the canary, and make rollback a single tested operation.

Continual learning is a spectrum

Continual learning updates a model as new data arrives; it does not require updating weights after every request.

• Full retraining fits a fresh model from a selected window. It is usually the safest default for tabular and ranking systems because it is reproducible and easy to compare and roll back.
• Warm-starting continues from the previous checkpoint. It can adapt faster, but carries old mistakes forward and makes results depend on the sequence of updates. Keep a fresh-from-scratch control.
• Incremental or online learning updates from small batches or events. It suits rapidly changing environments with fast labels, but adds ordering, duplicate events, late data, state recovery, concurrency, and rollback problems.

Recent-only updates can cause catastrophic forgetting, where useful behavior on rare or older cases disappears. Mix recent data with a protected historical slice and evaluate both recent and long-range holdouts.

Failure modes you can see first

The trigger flaps. Repeated jobs and rising compute spend indicate threshold noise. Add minimum sample sizes, cooldown, hysteresis, idempotent run IDs, and a concurrency limit.

The data is wrong but offline metrics improve. An implausibly large metric jump or a null rate suddenly reaching zero suggests leakage, bad joins, duplicate rows, or a broken transformation. Fail the data-quality gate and check data contracts and lineage.

Labels arrive late. A performance collapse at the newest boundary followed by later recovery often means unfinished outcomes. Delay evaluation to the label cutoff and distinguish “not observed yet” from “observed and negative.”

The candidate passes offline but fails in the canary. Missing features and timeouts within minutes suggest training-serving skew, inconsistent defaults, or excessive latency. Stop the canary, compare request-level features, and roll back. Training-serving skew is not fixed by training again.

The feedback loop narrows the system. If item A’s exposure rises from 35 percent to 82 percent while reported clicks rise, add exploration, preserve exact propensities, and evaluate coverage and new-item exposure. The model-generated dataset is not an unbiased report card.

The production pattern

Separate four decisions:

• Monitoring: is the signal real?
• Data validation: is the snapshot fit to use?
• Model evaluation: does the challenger meet quality and guardrail rules?
• Promotion: should the serving reference change?

Automate the first three when their checks are trustworthy. Automate promotion only when the evidence and rollback path are equally trustworthy. High-impact models may require human approval; lower-risk systems may use an online gate.

Retraining is worthwhile when the data-generating process changes faster than the model’s useful lifetime, labels are reliable, and a new model can be evaluated before harm accumulates. It is the wrong first move when the feature pipeline is broken, the apparent drift is a seasonal pattern, or too few new labels exist. Sometimes the right action is to keep the champion, fix the input, and wait.

Practice

The recommender’s click rate fell from 8.1 percent to 6.4 percent, but the event collector dropped 20 percent of click events on those days. Should retraining start, and which gate should stop a candidate built from that period?

Now suppose the data is valid. The challenger wins clicks but reduces catalog coverage from 42 percent to 24 percent. Is that a win? The predeclared product guardrail decides—not the most attractive dashboard number.

A question to carry forward

Retraining handles a model that is aging. But a feature pipeline can ship a bug at 1 a.m., making predictions garbage by 3 a.m. There is no time for a careful retrain and long experiment. First contain the damage, identify the change, and restore a known-good state.

That is incident response: what to do during the 3 a.m. ML page, when the model is failing now rather than merely aging.