Why a deployed model does not learn from a chat—and how fine-tuning can cause forgetting

At 3:12 a.m., a support bot answers a customer confidently: “All sales are final.”

That answer used to be correct only for a narrow case. The bot had handled refund questions well for months. Then the team fine-tuned it on a new batch of invoice examples. The invoice tests improved. The old refund behavior quietly fell apart.

This is the uncomfortable truth behind many “learning AI” demos: the deployed model did not learn from the conversation, and teaching it later may damage what it already knew. It is not a junior employee accumulating experience. It is a frozen mathematical function until somebody runs another training job.

The first problem is frozen weights. The second is catastrophic forgetting, the tendency for a model trained sequentially on new material to lose performance on earlier tasks. These are related, but they are not the same problem. Confusing them leads teams to fine-tune when they need retrieval, and to blame retrieval when a bad update has actually damaged the model.

A deployed model is a frozen function

A neural network’s weights are the numerical parameters it learned during training. They encode patterns such as:

• which words tend to appear together;
• how instructions map to actions;
• what visual features matter; and
• how pieces of a sentence influence one another.

You can think of a trained model as a function, written f_theta(x), where x is the input and theta is the current set of weights. Training changes theta. Inference, meaning the act of producing an answer, normally does not.

Once training ends, the deployed checkpoint is static. The serving system feeds it your prompt, runs the network forward, and returns tokens. It does not calculate a learning gradient or write a new checkpoint after the answer. The model may produce different wording because of sampling, or because you supplied different context, but it has not updated itself.

That is why a model has a knowledge cutoff, the latest point represented in its training data. Ask about an event after that date and the model has no built-in path to discover it. It may guess. It may sound impressively certain while guessing. Neither is learning.

A conversation can make this harder to notice. Your earlier messages sit in the current context, so the model can use them while generating the next answer. When the context disappears, that temporary adaptation disappears too. If an application stores a summary of your preferences and inserts it into future prompts, that is an external memory system. It can be useful, but the model’s weights have still not changed.

The two ways teams try to add knowledge

Suppose our Northstar support bot needs the current invoice policy. There are two common approaches.

Retrieval-augmented generation, usually shortened to RAG, retrieves relevant documents at query time and places them in the model’s context. The policy lives in a document store or search index. When the policy changes, the system updates that store. The model remains untouched. The RAG basics explain the moving parts in detail.

This is a good fit for facts that change:

• prices;
• product manuals;
• legal policies;
• inventory;
• internal procedures; and
• customer-specific records.

The answer can include the source document and its effective date. A model with a library card is not a model that has read the book, but that distinction is often exactly what keeps production systems safe.

Fine-tuning is additional training on selected examples. It changes the weights, either throughout the model or through a smaller trainable component such as an adapter. Fine-tuning can teach a model:

• a stable output style;
• a specialised classification boundary; or
• a repeated tool-use pattern.

It is also the route that creates the forgetting problem.

That distinction matters. RAG gives the model new evidence for one request. Fine-tuning changes the function that answers many future requests. The first is easier to update and roll back. The second can make a capability more automatic and cheaper at serving time, but it needs regression testing.

Why new training damages old behaviour

The weights are not a filing cabinet with one drawer labelled “refunds” and another labelled “invoices.” Knowledge and behaviour are distributed across many shared representations. The same internal features may help the model recognise a date, follow a policy exception, format a table, or decide that a request is unsafe.

During fine-tuning, gradient descent changes weights in the direction that reduces the training error. In simplified form, an update looks like this:

theta_new = theta_old - eta * gradient(L_new, theta_old)

Here, L_new is the loss on the new examples and eta is the learning rate, which controls the size of each step. The important detail is what is missing: there is no term telling the optimiser to preserve the old refund answers unless you deliberately provide one.

How interference appears

Imagine Northstar’s original held-out evaluation, meaning examples kept separate from training:

• 96 of 100 old refund questions answered correctly.
• 94 of 100 safety-escalation questions handled correctly.
• The new invoice task was not yet available.

The team fine-tunes on 4,000 invoice transcripts. On a separate held-out 100-example invoice evaluation, whose examples were never included in those 4,000 transcripts, the candidate reaches 92 of 100. That sounds like a successful update. But on the old evaluation, refunds fall to 63 of 100 and safety escalations fall to 70 of 100.

Those numbers are an illustrative failure shape, not a universal benchmark. The point is the trade: the training objective rewarded invoice performance and had no reason to protect the old tasks. An update that helps the model separate invoice fields can alter the shared representations used to interpret refund exceptions or safety language.

The old knowledge has not necessarily been physically erased from one neat location. The paths through the network that once produced the old answer have changed. The result is what matters operationally: the same prompt now produces a worse answer.

“Catastrophic” does not mean the model forgets every fact. It means performance on an earlier task can drop abruptly compared with the improvement on the new task.

Factors that affect severity

The severity depends on:

• the learning rate;
• the number and diversity of new examples;
• how much the tasks overlap;
• which parameters are trained; and
• whether old examples are included.

There is one more trap. Sometimes the old answer should change. If Northstar’s refund policy genuinely changed from 30 days to 60 days, a drop on an old policy test is not forgetting; it is an outdated test. Regression sets need versioned expectations. Otherwise a team can preserve obsolete behaviour and call it stability.

Why “just train on everything” is not a deployment strategy

The simplest way to avoid forgetting is to keep training on old and new data together. The old examples continue to exert pressure on the model, so an update cannot optimise only for the newest batch.

That is also expensive. Suppose the historical training mixture contains 100 billion tokens and a new day contributes 1 billion tokens. Rebuilding from the full mixture means processing roughly 101 billion tokens instead of only the new billion, even before accounting for repeated epochs, evaluation, infrastructure, and the operational delay. The comparison is illustrative, but the shape is real: history grows while the new information usually arrives in small increments.

Continual learning is the research and engineering problem of updating a model over successive data streams while preserving useful previous capabilities. The main approaches each protect against interference in a different way.

Common ways to reduce interference

Replay keeps a sample of old examples and mixes them into new training. Northstar might train on invoice examples alongside representative refund, escalation, and account-support cases. Replay works because the old loss remains present during the update.

Its costs are:

• data storage;
• sampling strategy;
• privacy review; and
• coverage.

A buffer containing only easy old examples will not protect the hard cases. Generated old examples can help, but they can also recycle and amplify the model’s existing mistakes.

Regularisation discourages important old parameters from moving too far. Elastic Weight Consolidation, or EWC, is a classic example: it estimates which parameters mattered for an earlier task and adds a penalty for changing them.

This is useful when there is a meaningful set of parameters worth protecting. It is less useful when the new task genuinely needs those same parameters, because preserving the past and fitting the present then pull in opposite directions.

Parameter isolation gives new learning its own parameters. LoRA, or Low-Rank Adaptation, represents a trainable update with small low-rank matrices while the base model stays frozen. A separate adapter for each department can prevent the invoice update from directly rewriting the base model.

But LoRA is not a magic anti-forgetting switch. If one adapter is trained sequentially, it can forget within that adapter. If many adapters exist, the system needs reliable routing. If an adapter is merged into the base, the original interference returns in a different form.

Retrieval avoids changing the weights altogether. It is often the right production answer for Northstar’s current policy documents because the documents are the source of truth, not the model’s latent memory.

Retrieval still has failure modes:

• stale indexes;
• missed documents;
• poor chunking;
• access-control mistakes;
• prompt injection; and
• context that is too long for the model to use reliably.

It sidesteps catastrophic forgetting; it does not make information retrieval magically correct. Long context has its own trade-offs.

The safest systems usually combine these ideas rather than choosing one forever: retrieval for volatile facts, a controlled fine-tune for stable behaviour, replay or regression tests for old capabilities, and a rollback path for every model update.

The strongest objection is partly right

The fair objection is that modern large models do not collapse after every fine-tune. They often have spare capacity. A small, carefully chosen adapter can teach a narrow behaviour while preserving most general performance. New examples can even improve old tasks when the tasks are related. Humans also learn continuously without losing every earlier skill.

All true.

The mistake is turning “forgetting is not inevitable” into “forgetting does not need testing.” A model can preserve broad chat ability while losing the one narrow behaviour your business depends on. A support model may still write elegant paragraphs while misapplying the refund exception that costs money. Aggregate quality can look fine while a critical slice rots.

The human comparison also hides an important engineering difference. People have episodic memory, rehearsal, selective attention, and the ability to retrieve a specific past experience. A deployed model does not automatically maintain those systems. If your application builds them around the model, that is good architecture. It is not evidence that the weights learned from the conversation.

RAG has a fair objection too. It adds:

• retrieval work;
• network calls;
• prompt tokens; and
• another system that can fail.

For a stable skill such as “return this schema and call the inventory tool,” putting the behaviour into a well-tested fine-tune may be faster and cheaper than pasting instructions and examples into every request. The right comparison is not “RAG good, fine-tuning bad.” It is “where should this changing information live, and what must remain stable?”

The fine-tuning versus RAG decision becomes much clearer when phrased that way:

• A changing fact belongs in a source that can be updated and cited.
• A user preference belongs in controlled external memory with deletion and consent.
• A stable capability or output behaviour may justify fine-tuning.
• A live calculation belongs in a tool or database, not in model weights.

What to do on Monday morning

Classify the update

Start by writing down what “learn” means for the proposed update. Put each request into one of three buckets:

• new facts;
• new behaviour; or
• user-specific state.

This small classification prevents a surprising amount of bad fine-tuning.

Test before training

Then build a regression set before changing the model. For a small production system, begin with 100 old prompts that represent real traffic and known edge cases. Add 50 prompts for the new requirement. Keep a held-out portion out of training.

Record more than a single quality score:

• correctness;
• refusal behaviour;
• citations;
• tool calls;
• latency; and
• token usage.

All matter.

Route changing facts through retrieval

Route changing facts through retrieval first. Store:

• document versions;
• effective dates;
• source ownership; and
• access rules.

Test the complete path, including re-indexing and cache invalidation. If the updated policy is present in the retrieved context but the answer is still stale, you have a retrieval or instruction problem, not necessarily catastrophic forgetting.

Train and evaluate the candidate

For a fine-tune, preserve old examples in the training mix or use another explicit preservation method. Train a candidate checkpoint or separate adapter. Run the old and new evaluations after every meaningful candidate.

Set a maximum acceptable regression before you look at the results. Otherwise a shiny new score will persuade the team to explain away the old damage.

Deploy with rollback

Finally, deploy behind a canary and keep the previous checkpoint available for immediate rollback. Monitor old-task slices separately from new-task slices.

The first symptom of forgetting is often not a dramatic outage. It is a familiar question receiving a confident, slightly wrong answer while the newly trained demo looks excellent. That is exactly when a regression dashboard earns its keep. The surrounding operational discipline belongs to LLMOps, not to wishful prompting.

The practical mental model is simple: use weights for durable capabilities, retrieval for changing knowledge, and external memory for user state. A model that can keep learning without damaging its old contract would be extraordinary. Until that is reliable, the responsible system is not the one that updates itself most eagerly. It is the one that knows where change belongs and can prove what it broke.