Pretraining data: curation, dedup & quality

A language model is, almost literally, what it eats. Everything it knows and every habit it has comes from the text it was trained on — so the quality of that text is a first-class lever, right next to the architecture. And the raw material is grim: a web crawl is mostly spam, boilerplate, navigation menus, near-duplicate reposts, and machine-generated filler. Pour that in unfiltered and you get a fluent model with no taste.

The unglamorous pipeline that turns petabytes of raw crawl into a clean, deduplicated training set is a big part of why two models with the same architecture and parameter count can be worlds apart. This lesson walks that pipeline.

Where the tokens come from

Frontier pretraining sets are assembled from a few broad sources:

• Web crawl — the bulk, usually from Common Crawl (petabytes of raw HTML scraped over years). Enormous, and mostly low quality.
• Code — GitHub and similar. Punches above its weight: code teaches structure, reasoning, and long-range consistency, and it helps even on non-code tasks.
• Books and long-form — high-quality, coherent, long-context prose.
• Reference — Wikipedia, papers, curated Q&A.
• Synthetic — increasingly, text generated by other models (more on this below).

The web is where the volume is and where the cleaning problem lives. Everything that follows is mostly about making the web crawl usable.

The curation funnel

Curation is a funnel: each stage throws away more than it keeps, and what survives is a small, dense fraction of the input.

Extraction and language ID. Strip HTML to text, drop the nav/ads/boilerplate, and keep only the languages you are training on. Already most of the raw bytes are gone.

Quality filtering. Decide whether a document reads like something worth learning from. Two styles, usually combined: cheap heuristics (reject pages that are too short, mostly symbols, keyword-stuffed, or missing normal stopword patterns) and a learned classifier that scores how much a page resembles known-good reference text. FineWeb-Edu, for instance, trained a classifier to keep “educational” web pages and saw downstream gains from that filter alone.

Deduplication — the quiet high-ROI step

The single highest-leverage cleaning step is usually deduplication, because the web is staggeringly repetitive: the same article is reposted across hundreds of sites, each wrapped in different navigation and ads. Removing duplicates buys three things at once:

• Less memorization / better privacy — text seen many times is more likely to be regurgitated verbatim, including any personal data in it.
• No train-test contamination — duplicates of benchmark questions hiding in the crawl would otherwise inflate eval scores (see decontamination below).
• Cheaper, better training — you do not pay FLOPs to read the same paragraph a thousand times, and heavy duplication can actively hurt quality.

Exact-match dedup is easy (hash each document). The hard, important case is near-duplicates — same content, different boilerplate. The standard tool is MinHash + LSH: represent each document as a set of overlapping word shingles and measure the Jaccard similarity (shared shingles over total). High overlap = near-duplicate. (Semantic dedup, e.g. SemDedup, goes further and clusters by embedding similarity to catch paraphrases.)

def shingles(text, k=4):                      # overlapping k-word sequences
    t = text.lower().split()
    return set(tuple(t[i:i+k]) for i in range(len(t) - k + 1))

def jaccard(a, b):
    A, B = shingles(a), shingles(b)
    return len(A & B) / len(A | B)            # shared / total

article = "the central bank raised interest rates by half a point on tuesday citing inflation"
copy_a  = article + " all rights reserved subscribe now"      # same article, site A
copy_b  = "home about contact " + article + " cookie policy"  # same article, site B
other   = "researchers trained a new vision model on a billion labeled images"

print("same article, different boilerplate:", round(jaccard(copy_a, copy_b), 3))
print("article vs unrelated doc          :", round(jaccard(article, other), 3))
print("flag as duplicate at threshold 0.5:", jaccard(copy_a, copy_b) >= 0.5,
      "/", jaccard(article, other) >= 0.5)

same article, different boilerplate: 0.524
article vs unrelated doc          : 0.0
flag as duplicate at threshold 0.5: True / False

The two reposts share most of their shingles (0.52) and get flagged as the same content despite different wrappers, while the unrelated document scores 0. Run this across billions of documents with MinHash signatures (so you never compute the full pairwise comparison) and the duplicate reposts collapse to one copy.

Decontamination and safety

Decontamination removes any copies of evaluation benchmarks’ test questions from the training data. Skip it and your model may “ace” a benchmark simply because it memorised the answers during pretraining — a measurement artifact, not a capability. Safety/PII filtering strips or down-weights toxic content and personal data so the model is less likely to emit it.

The mixture is the model

Two sets of the same size can produce very different models, because what you mix in determines what the model can do. Recipes deliberately up-weight high-value sources — code, books, reference text — above their natural web proportion, because each disproportionately improves reasoning and coherence. The mixture is a tuned recipe, not an accident.

In one breath

• A model is what it eats; raw web crawl is mostly junk, so curation is a first-class lever next to the architecture.
• Pretraining data comes from web crawl (the bulk, e.g. Common Crawl), code, books, reference, and increasingly synthetic text.
• The curation funnel — extract + language-ID, quality-filter (heuristics + classifier), deduplicate, decontaminate + safety — keeps only ~10–15% of the raw input.
• Dedup is the quiet high-ROI step: it cuts memorization, prevents train-test contamination, and saves compute; near-dups are caught with MinHash/LSH on shingle Jaccard similarity (0.52 for two reposts of one article vs 0 for unrelated text).
• The mixture decides capability — up-weighting code/books/reference — and the modern consensus is quality over quantity (Chinchilla, the phi “textbooks” result), with synthetic data the answer to the looming data wall.

Quick check

Next

You have seen how the training data is built; pretraining covers the objective that turns it into weights, and scaling laws explain how data and parameters should grow together. For shaping behaviour after pretraining, head to alignment.