Don't auto-ship retrained models: collapse, feedback, and the challenger gate

At 03:00, the retraining job finishes. Its headline metric is better. The deployment pipeline promotes it automatically. By breakfast, the support queue is full of complaints, a valuable user segment has stopped engaging, and nobody can quite say which data changed.

Suppose a news app’s old model gets an eight percent click-through rate. The new model scores better offline: its ranking AUC, a metric asking whether useful items tend to rank above unhelpful ones, rises from 0.84 to 0.86. The team sends one percent of traffic to it. Clicks rise by 0.2 percentage points, but reads longer than 30 seconds fall from 5.0 percent to 4.3 percent. Complaints rise from 0.4 percent to 0.6 percent.

That is not a successful retraining run. It is a regression wearing a lab coat.

The safe position is simple: automate the training, but make promotion earn its way through a gate. The live model is the champion. A freshly trained challenger is only a candidate until it passes:

• data checks;
• offline tests;
• a shadow run;
• a controlled canary.

If it fails before promotion, the intended action is to keep the champion.

Champion-challenger automates retraining while limiting the blast radius of regressions. A canary and a tested rollback path reduce risk; they do not guarantee zero user exposure or zero regressions. Rollback can fail, too, so test it before the 3 a.m. incident, not during it.

Your model changes the data it learns from

Exposure shapes the training set

A model in production is not merely observing the world. It is changing which parts of the world become visible.

Consider the news app. Its champion ranks stories on every home page. The logging system records impressions and clicks. But a click means “this person clicked after seeing this story,” not “this person would have clicked this story if it had been shown.” That missing counterfactual is the heart of the problem.

If the champion shows story A 900,000 times and story B 100,000 times, the next training set contains far more evidence about A. Perhaps A really is better. Perhaps the champion already favored A, so B never got enough exposure to prove itself. The next model sees the result of the old policy and can mistake that result for user preference.

This is exposure bias, meaning that the training data overrepresents whatever the previous model chose to expose. Researchers have documented this kind of feedback loop in recommender systems. It is not a philosophical concern. It changes the distribution of the next training set.

Correct for exposure

A practical defense is to log the propensity score, the conditional probability that the logging policy, meaning the policy that actually made the serving decision, selected the observed action or slate given the request context.

An action is the choice being evaluated, such as showing story B. A slate is the whole ranked list. For ranked recommendations, the relevant quantity can be the probability of the observed slate under the logging policy, including position and exposure effects where they matter. An item’s marginal exposure probability is not always enough.

Log the context, the selected action or slate, the logging-policy version, and the conditional propensity used for that decision. The policy version matters because a probability produced by yesterday’s exploration rules is not necessarily valid for an event served by today’s rules.

To evaluate a target policy, meaning the policy you want to compare offline, weight each observed outcome by the target policy’s propensity divided by the logging policy’s propensity for that observed action or slate:

w = target propensity / logging propensity

The familiar 1 / p rule is only a special case in which the target policy would choose the observed action or slate with probability one. In a simple one-item example, if B had logging propensity 0.1 and the target policy would choose B with certainty, the observation gets weight 1 / 0.1 = 10. If A had logging propensity 0.9 and the target policy would choose A with certainty, its weight is 1 / 0.9, or about 1.1. With a stochastic target policy, the numerator is its actual probability, not automatically one.

This correction requires positivity, also called overlap: every action or slate the target policy might choose must have nonzero probability under the logging policy in the same kind of context. If B was never shown to users like this, no weighting trick can recover its unseen outcome. Accurate propensity estimates and accurate logging are required as well.

Very small logging propensities create high variance. A handful of unusual clicks can dominate the estimate. Clipping weights caps those extreme values; self-normalization rescales the weighted estimate. Both can reduce variance, but they generally trade some bias or a changed finite-sample estimand for that stability. Report the choice instead of presenting a clipped estimate as if it were exact.

You usually also need a deliberate exploration slice, in which the system occasionally tests reasonable alternatives. Exploration is not “add randomness and hope.” It is an investment in future evidence. Without it, the target policy may ask questions the logs have no way to answer.

Drift starts an investigation

This is also why drift is a trigger, not a verdict. A change in feature distribution may mean the audience changed, or it may mean an upstream parser started emitting empty values. Drift monitoring can tell you that the world or the pipeline moved. It cannot tell you that the newly trained model deserves production traffic.

Model collapse is a different, deeper failure

Feedback loops and model collapse are often lumped together. They are related, but they are not the same failure.

A feedback loop happens when a model’s decisions affect the observations used to train its successor. Model collapse happens when model-generated training examples are recursively reused across generations, especially when generated data replaces or dominates independent real data.

The mechanism is straightforward. Start with a real distribution of text, images, or other examples. A model learns an approximation of it and generates new examples. Those examples contain the model’s errors and usually represent common patterns more heavily than rare ones. If the next model trains mainly on that generated set, it treats the errors as evidence. Its output becomes the next training set, and the distortion compounds.

Rare events disappear first because they were already unlikely to be generated. Once they are underrepresented in one generation, the next model has even less chance of learning them. The center of the distribution may still look polished while the tails quietly vanish. That is why a model can appear fluent, accurate, or “on brand” while becoming brittle on unusual inputs.

The paper by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal describes this recursive failure and the loss of distribution tails in The Curse of Recursion: Training on Generated Data Makes Models Forget.

For the news app, a recommendation is model output, but a user’s later click is a real user event collected under the champion’s exposure policy. The successor is learning from outcomes collected under its own serving decisions; the logged recommendation itself is not the label. That is exposure feedback, not automatically model collapse.

This differs from directly training on model-generated articles, examples, preferences, or labels. Recursively reusing generated articles or other generated examples across generations can cause model collapse, especially if they replace or dominate real data. Generated preferences or labels do not automatically mean collapse; they can still reinforce the teacher model’s errors and need independent validation.

The quieter traps: forgetting and ordinary bugs

Incremental retraining can also cause catastrophic forgetting, the loss of older knowledge when updates focus too heavily on recent examples. A model trained only on the last seven days may improve on breaking news and new users while degrading for older users, minority languages, or less popular topics. The gradient update has no reason to preserve behavior for examples it no longer sees.

A replay mixture of older data, recent data, and carefully chosen rare slices helps. So does evaluating every candidate against both a recent temporal holdout and a stable historical set. “The latest batch looks good” is not a model quality strategy.

And many retraining disasters are less exotic. A label join can shift by one day. A feature can be null in production but imputed during training. A backfill can leak future information into an offline split. A library upgrade can change tokenization or ranking behavior. The first symptom is often a candidate that looks dramatically better offline and then produces malformed outputs, empty features, or a sudden online metric drop.

The challenger gate is a sequence of evidence

Champion-challenger is not a ceremony in a model registry. It is a production decision with explicit failure behavior.

The champion continues serving users. The challenger receives the same inputs and is tested without being trusted. Promotion happens only when the challenger meets predeclared requirements. A useful mental model is:

promote = valid data and offline pass and shadow pass and canary pass

Each term exists because it catches a different class of mistake.

First, prove that the candidate is what you think it is

Record the candidate’s:

• training-data version;
• data cutoff;
• label window;
• feature schema;
• code revision;
• dependency versions;
• model artifact checksum;
• fraction of synthetic or weakly labelled data.

Store the model’s identity with every prediction.

This makes a candidate reproducible. It also makes comparison meaningful. If the news app’s challenger used data through Tuesday while the evaluation features accidentally included Wednesday’s popularity counts, the apparent improvement is leakage, not learning.

Then test more than the average

Use a temporal holdout: train on data available before a cutoff and evaluate on later data. Random splits can place near-duplicate events or future aggregates on both sides, producing a wonderfully clean lie.

Evaluate the news model on at least three views:

• the full holdout, for broad performance;
• a recent window, for current conditions;
• important slices such as new users, returning users, languages, devices, and low-frequency topics.

Track business outcomes as well as convenient model metrics. In our example, the challenger might improve AUC and ordinary clicks while failing an absolute floor for 30-second reads or increasing complaint rates. It should not pass because one aggregate number went up.

Set both relative and absolute rules. “Beat the champion by 0.1 percent” is not enough if the champion itself is unacceptable. A candidate that improves a bad model can still be bad.

Do not confuse a tiny difference with a real win

Suppose the champion gets eight percent clicks and the challenger gets 8.2 percent in 100,000 impressions per arm. That is 200 additional clicks. Under a simple independent-rate calculation, the standard error of the difference is roughly 0.12 percentage points. The observed gain is suggestive, but not strong evidence on its own. Repeated impressions from the same users and other correlations can make the true uncertainty larger.

Choose the following before looking at the result:

• the minimum useful improvement;
• the sample size;
• the evaluation window;
• the stopping rule.

Otherwise the pipeline will eventually stop on the first attractive fluctuation.

Shadow the challenger

In a shadow deployment, the challenger receives a copy of production requests, but its outputs are discarded. Compare its output with the champion on:

• output shape;
• scores;
• timeouts;
• error rate;
• resource use;
• latency.

Shadowing catches failures that offline tests often miss:

• a feature service returns a different schema at runtime;
• the candidate needs twice the memory;
• a rare input produces invalid output;
• p99 latency becomes too high for the request budget;
• a dependency is unavailable in the serving environment.

Shadowing does not prove that users will like the new model. The challenger did not influence what users saw, so it did not create the feedback it is meant to replace.

Canary the behavior

A canary sends a small, controlled slice of real traffic to the challenger. It deliberately exposes those users to the challenger; it is not a zero-exposure test. For the news app, one percent of one million daily impressions is about 10,000 impressions. That can reveal a severe regression quickly, but it will not reliably detect a subtle improvement. Traffic assignment should be randomized and stable enough that the same user does not bounce between models every request.

Watch:

• quality;
• complaints;
• diversity;
• latency;
• errors;
• spend.

Set automatic rollback thresholds before the canary begins. Roll back to the known champion, and verify that routing actually returned to it; rollback itself can fail. Do not launch another retraining job and hope it is healthier.

Offline, shadow, and canary checks can still miss a regression. Labels may arrive later, an important outcome may not be measured, or a harmed segment may be too small in the canary to produce a reliable signal. A canary limits the number of people exposed to a bad candidate. It does not make the candidate safe by definition.

Finally, respect label delay. If meaningful retention or complaint labels arrive seven days later, a two-hour canary cannot prove those outcomes. Keep the challenger in a pending state until the labels mature, or use a clearly marked proxy and accept that the decision is provisional.

The strongest objection is real: champions can rot

The fair objection is that a gate slows adaptation. If news changes hourly, an old champion can be worse than an imperfect challenger. Requiring a human approval for every candidate does not scale, and a permanently cautious system can become a permanently stale system.

The answer is not to remove the gate. It is to make the gate fast, automated, and honest about what it knows. Offline checks can run on every training completion. Shadow traffic can run continuously. A low-risk canary can promote automatically when its predefined floors hold. High-impact models can require human review.

Use a scheduled retraining floor as a backstop, but do not treat a calendar as evidence. Trigger work from:

• drift;
• label-based performance degradation;
• a known business change.

If a champion reaches a maximum age and no challenger has passed, raise an incident and keep the champion or use a simpler reviewed fallback. Shipping an unproven candidate merely because the calendar says Tuesday is not freshness; it is surrender.

There is a cost:

• Shadow inference consumes compute.
• Canarying requires randomization, monitoring, delayed-label handling, and a tested rollback path.

For a disposable internal classifier with no customer impact, direct deployment with a basic health check may be a rational trade. For a model deciding what millions of people see, spend, or receive, the extra machinery is cheap insurance.

A gate is also useless when there is no trustworthy outcome signal. If nobody can label success and every proxy is easily gamed, do not pretend that a green dashboard proves quality. Improve the evaluation data first.

What to do on Monday morning

Start with the model that pages you most often. Do not redesign the whole platform.

1. Make the current champion immutable. Record its artifact ID, feature definition, training cutoff, and rollback command. Test the rollback while the system is healthy.

2. Write an acceptance contract for the challenger. For the news app, it might require:

   • no more than a 0.2 percentage-point increase in complaints;
   • no drop below 4.8 percent in long reads;
   • bounded latency;
   • no material loss for new users.

   Those values are product decisions, not universal constants.

3. Make every training run emit a candidate manifest containing:

   • data versions;
   • label window;
   • code revision;
   • feature schema;
   • provenance;
   • metrics;
   • resource requirements.

   A model without this information is not a candidate. It is an archaeological object.

4. Fix the event log. Store:

   • the model ID;
   • request ID;
   • selected items;
   • scores;
   • policy version;
   • exposure probability where available;
   • eventual outcome.

   Without the serving context, later analysts cannot tell preference from exposure.

5. Build a temporal holdout and a recent evaluation slice. Add absolute floors for the outcomes that matter. Keep older examples in the evaluation set so incremental training cannot hide forgetting.

6. Run the candidate in shadow, then send a deliberately small randomized canary. Automate rollback for errors, latency, and severe business-metric regressions. Leave promotion pending when labels have not matured.

The first useful dashboard is not “latest model accuracy.” It is a decision record: which candidate challenged which champion, on what data, under which rules, with what result.

Symptoms worth investigating first

First symptom	Likely cause	First check
Offline metrics improve, but canary quality falls	Leakage, exposure bias, or a train-serving mismatch	Rebuild the temporal split and compare runtime feature values
Rare topics or items disappear after several updates	Feedback loop, excessive exploitation, or synthetic-data contamination	Inspect exposure counts and record provenance
Candidates remain pending indefinitely	Labels arrive late or the traffic slice is too small	Check the label watermark and the planned sample size
Shadow p99 latency or error rate jumps	Dependency, schema, or artifact incompatibility	Compare feature schemas and serving-resource traces

The point of automated retraining is not to make a new model live every time new data appears. It is to make improvement cheap without making regression inevitable. Keep the champion live until the challenger has earned the traffic.

Related lessons

• Retraining and continual learning — scheduled versus trigger-based updates
• Drift and monitoring — the signals that justify a new candidate
• Model registry and promotion — making the gate enforceable rather than ceremonial