What's the difference between full retraining, incremental (warm-start) training, and continual online learning?
Full retraining trains a fresh model from scratch on the latest data window, giving the cleanest result but at the highest cost and slowest cadence. Incremental or warm-start training continues from existing weights on new data, which is cheaper and faster but can accumulate drift and forgetting. Continual online learning updates the model continuously from a live stream for maximum freshness, at the cost of stability, harder evaluation, and vulnerability to bad or poisoned data.
How to think about it
The short answer
Three update strategies on a freshness-vs-stability spectrum: full retraining (train from scratch on the latest data), incremental / warm-start (continue from current weights on new data), and continual online learning (update continuously from a live stream). Freshness increases left to right; so does operational risk.
The three approaches
- Full retrain: rebuild the model on a rolling data window. Cleanest, most reproducible result; easiest to evaluate. Costliest and slowest, so cadence is days/weeks. The default for most teams.
- Incremental / warm-start (stateful): initialize from the existing checkpoint and train on just the new data. Much cheaper and faster, enabling frequent updates. Risk: it can accumulate drift and catastrophic forgetting, and bugs compound across generations since you never reset.
- Continual online learning: the model updates per-batch or per-event from production data, maximizing freshness for fast-moving domains (recommendations, ads, fraud). But evaluation is hard, it’s unstable, and it’s highly exposed to poisoned or anomalous data that immediately bends the live model.
Concrete example
A news-feed ranker uses online updates to react to breaking topics within minutes. A credit-risk model uses full retraining monthly because stability, auditability, and regulatory reproducibility matter more than minute-level freshness. A mid-size CTR model might warm-start nightly to cut compute while occasionally doing a full retrain to reset accumulated drift.
Common follow-up / trap
Interviewers probe: “Why not always do online learning — it’s the freshest?” Because freshness trades against stability, evaluation rigor, and security: an attacker feeding crafted feedback can poison a continually-learning model in real time, and a single bad data batch hits users instantly. The trap is treating online learning as strictly superior. The mature answer pairs the strategy to the domain’s volatility and risk, and notes that even online systems typically periodically full-retrain to reset drift and forgetting.