Synthetic data and model collapse
How generated examples become useful training material, and how recursive reuse can erase the rare cases your model needs.
What you'll learn
- How synthetic data supports instruction tuning, distillation, rare-class coverage, evaluation, and privacy work
- Why generate-then-filter works only when the verifier is stronger than the generator
- How recursive training on sampled synthetic data causes model collapse
- Why replacing real data is more dangerous than accumulating synthetic data beside it
- Which diversity, provenance, and contamination checks catch narrowing early
Before you start
At 3:07 a.m., your support router starts sending database-outage tickets to the “general question” queue.
The aggregate accuracy dashboard still says 96.8 percent. That number is not lying. It is merely being unhelpful. Most tickets are ordinary password resets and billing questions. The rare database-corruption cases are only 0.5 percent of traffic, so a model can become worse at them while its overall score improves.
Your team has 100,000 real tickets. Only 500 describe the obscure failure modes that matter during an incident. You need more examples, but asking humans to write 50,000 realistic outage reports is expensive and slow.
A language model can write them in an afternoon.
That is synthetic data: data produced by a program, simulator, or model rather than directly collected from the process you care about. It can be extremely useful. It can also quietly remove the long tail of reality if you feed it back into training without safeguards.
The useful part and the dangerous part come from the same fact: generated data is a sample from a model’s beliefs about the world.
The legitimate uses
Synthetic data supports several jobs, but each has a different definition of “correct”:
- Instruction tuning: Generate varied examples such as “A deployment fails after a schema migration. Route this ticket and explain the evidence.” This adds coverage across formats, languages, and difficult wording, but fluent variations of an easy case do not teach edge cases.
- Distillation: A smaller student model can learn generated answers, rankings, or tool traces from a larger teacher. It inherits the teacher’s blind spots, though: ten paraphrases of one misrouted outage are ten copies of one mistake.
- Rare-class coverage: Generate candidates for known database failures rather than imitating the common 99.5 percent of traffic. Verify that symptoms, labels, and required fields are consistent and that examples are not duplicates. Calling a common ticket “rare” does not make it useful.
- Evaluation: Ask a model to propose confusing or adversarial cases, then independently check their answers. A case whose question and answer came from the same model is a rehearsal, not an independent exam.
- Privacy work: Generated records can reduce handling of raw customer text, but they are not automatically anonymized. A generator can memorize names, identifiers, or ticket fragments; search for them, test canary records, and use formal privacy methods when needed.
Generate first. Filter second.
The generator supplies breadth. The verifier decides what earns a place in training.
That division of labor is the central pattern.
Verification should match the task:
- Programmatic checks are strongest when truth is executable: compile code, run a query, validate JSON, compare calculations, or execute a tool call in a sandbox.
- Rules catch cheap domain errors, such as a database-outage label without any deployment, migration, connection, schema, or known incident signature.
- Rubric or LLM scoring can rank candidates for relevance and completeness, but a judge from the same model family as the generator may share its errors.
- Human review is necessary when correctness is semantic or consequences are high. Review a measured sample, especially from accepted records and rare modes.
A generator can produce 20,000 plausible tickets; the verifier determines whether 3,000 are useful. Improving the verifier changes the training signal more directly than making the generator more creative.
What model collapse actually is
Model collapse is a possible failure mode in which recursive training on model-generated data progressively loses information about parts of the original distribution, often beginning with rare modes.
A distribution is the frequency of possible examples. A mode is a recognizable concentration, such as “password reset” or “database corruption after a schema migration.” The tail contains low-frequency cases.
Imagine 100 rare modes in your support data. Each mode accounts for 0.01 percent of examples. A synthetic batch of 10,000 tickets expects:
10,000 × 0.0001 = 1
example from each mode.
The chance that a particular mode appears zero times is approximately:
(1 - 0.0001)^10,000 ≈ 0.368
So each mode has about a 36.8 percent chance of being absent from that batch. The batch can contain about 100 rare tickets in total while still missing many specific modes. “The rare class is present” is not the same as “the rare distribution is covered.”
Train the next model on that batch and it cannot learn a mode it never saw. Modes represented by one untypical or incorrect sample also have weak evidence.
The model therefore tends to allocate more probability to common, easy-to-reproduce patterns. In a specified representation—such as mode counts or distances in a fixed embedding space—samples may show reduced diversity and rare-mode coverage. For text, “variance” has no single meaning until you name the representation and statistic.
Sampling from that model produces data from an already narrowed distribution. Fit again, and the missing modes cannot spontaneously reappear because the new source contains no evidence for them.
The causal loop is:
- finite sampling omits or undercounts low-probability modes;
- model fitting turns those omissions into a changed distribution;
- the changed distribution produces fewer unusual examples;
- recursive training treats those fewer examples as reality.
This is not an inevitable property of all synthetic data. A perfect generator, sufficiently large samples, retained real data, and coverage-aware training can avoid the same failure. In practice, generators have errors, samples are finite, and pipelines often lose provenance.
The danger is greatest when synthetic data replaces real data.
Suppose the original 100,000-ticket set contains 500 rare outage tickets, or 0.5 percent. After several imperfect steps, assume a generator produces only 0.2 percent rare tickets.
A replacement corpus of 100,000 generated tickets contains about 200 rare tickets on average: 60 percent fewer than the original.
If you retain the original set and add 20,000 generated tickets, the combined corpus has about 540 rare tickets: 500 real plus an expected 40 synthetic. The rare rate falls to about 0.45 percent, but the original evidence remains.
This mixture is not automatically safe—synthetic data can overwhelm real data if its volume or sampling weight is too large—but one generation step is less able to erase the anchor.
Research on recursive training, including results from Shumailov and colleagues, describes this as loss of the tails and then of the original distribution. The practical lesson is simple: a generated corpus must never become the only memory of the world.
Detect narrowing before users do
Compare every synthetic batch with a fixed, stratified probe set of real data. Track:
- Mode and label coverage: Count incident type, language, customer segment, severity, and meaningful rare subtypes. A stable category count can hide disappearing difficult cases.
- Distinctness:
distinct-ncounts unique token sequences of lengthndivided by the total number of such sequences. It catches template collapse, but not semantic duplicates. - Entropy: Monitor how spread out categorical labels are. Use it as a trend, not a target; random nonsense can have high entropy.
- Semantic coverage: Embed real and synthetic examples with the same fixed model and check whether real examples have nearby synthetic neighbors. Embedding metrics can also hide errors, so combine them with other checks.
- Provenance: Record the generator, prompt or program version, sampling settings, verifier version, source snapshot, and acceptance reason.
Keep sample size and tokenization fixed across rounds, and inspect uncertainty when counts are small. A drop in rare-mode coverage is more informative than a single diversity score.
Synthetic evaluation data can contaminate the exam
Evaluation contamination does not require copying an exact benchmark question. If synthetic eval cases come from the same tickets used to train the router, the model may recognize shared wording, templates, or labels. The score then measures pattern reproduction rather than diagnostic skill.
Keep a genuinely independent real holdout. Useful protections include human-authored cases never exposed to the generator, temporal splits, exact and near-duplicate checks, withheld source documents, and a private real-data probe for synthetic development benchmarks.
Synthetic evals remain useful for regression testing. Preserve their lineage and call them development or challenge sets, not independent evidence.
Choosing synthetic data honestly
Use synthetic data when it improves measurable coverage, speed, or cost. Prefer real or authoritative sources when correctness cannot be independently checked.
| Need | Generate and filter | Realistic alternative | Deciding axis |
|---|---|---|---|
| Instruction examples | Cheap breadth and format variation | Human-written examples | Cost versus subtle correctness |
| Rare incidents | Targeted candidates for known modes | More incident collection or reweighting | Can the rare label be verified? |
| Teacher behavior | Many outputs or rankings | Human labels or teacher logits | Teacher reliability |
| Edge-case tests | Fast candidate discovery | Private human-authored holdout | Discovery or final evidence? |
| Less customer-text exposure | Controlled substitute records | De-identification or differential privacy | Threat model |
| Ground-truth facts | Usually poor without executable checks | Authoritative retrieval | Can correctness be checked? |
The hard limit: the verifier sets the ceiling
A synthetic dataset can only be as good as its verifier. If the filter checks grammar instead of whether a database failure is possible, the error enters training. If an LLM judge rewards confidence, the dataset teaches confidence.
Measure the verifier with blinded expert labels on random accepted and rejected candidates. Estimate precision among accepted records and recall where possible. Use executable checks for executable truth, independent references for factual claims, and human review where semantics matter. Preserve rejected examples: they reveal how the generator fails.
Never discard original real data merely because synthetic data is larger, cleaner, or easier to access.
What to remember
- Synthetic data expands coverage; it does not replace reality.
- Generate broadly, then verify with execution, rules, references, and sampled human review.
- Model collapse begins when finite sampling loses rare modes and recursive training turns those omissions into the next distribution.
- Retain real anchors, control synthetic weights, and monitor coverage, diversity, provenance, contamination, and real-data performance.
Quick check
Practice this in an interview
All questionsData leakage happens when information that would not be available at prediction time influences model training, producing overly optimistic evaluation metrics that collapse in production. Common sources include fitting preprocessors on the full dataset, including target-derived features, and using future data in time-series pipelines.
Production degradation stems from distributional shift between training and serving data, upstream pipeline changes, feedback loops, and the static nature of a trained model against a changing world. Offline evaluation on a held-out slice of historical data cannot simulate these dynamics.
GANs train a generator and discriminator adversarially to produce sharp samples but suffer from unstable training and mode collapse. VAEs optimise a tractable evidence lower bound for principled probability modelling but generate blurry samples. Diffusion models iteratively denoise from Gaussian noise, achieving state-of-the-art sample quality and diversity at the cost of slow sampling.
Freeze a representative test set before experimentation, then use only development data for preprocessing, feature selection, tuning, and model comparison. Fit every learned transformation inside each cross-validation fold, choose the model on development data, and evaluate the frozen choice on the test set once, using group or forward-chaining splits when rows are related or time-ordered.