You are reviewing a pretraining corpus assembled from web, code, and licensed data. How would you handle deduplication, quality filtering, benchmark contamination, licensing, personally identifiable information, and the trade-off between differential privacy and model utility?

How to think about it

I would make the corpus auditable before making it large: track provenance and rights, remove exact and near duplicates, filter quality and PII, quarantine evaluation material, and version every decision. I would use differential privacy only against a defined threat model, because stronger privacy can reduce utility and does not replace consent, licensing, or careful data minimization.

Why this is harder than “clean the data”

The interviewer is probing whether you see a corpus as a governed product, not a pile of text files.

Every retained document should have a source identifier, retrieval date, data type, license evidence, jurisdiction where relevant, transformation history, and deletion path. A source that cannot be explained six months later is not really part of a controlled training set. It is a future incident report wearing a .jsonl extension.

I would put the controls into a reproducible pipeline:

Concern	Practical control	Evidence to retain
Duplication	Exact hashes, then near-duplicate detection	Cluster IDs and retained representative
Quality	Language, boilerplate, spam, safety, and domain-specific checks	Filter scores and sampled decisions
Contamination	Compare against evaluation items and paraphrased solutions	Match reports and exclusions
Licensing	Source-level rights inventory and policy gates	License text, URL, timestamp, approval
PII	Detection, redaction or removal, secret scanning	Detection categories and deletion records
Privacy	Threat model, privacy accounting, utility evaluation	Privacy parameters and model comparisons

Deduplication

Duplicates waste tokens and increase memorization. If the same article appears 500 times through syndication, the optimizer sees it as 500 votes. That can distort the data distribution and make the model unusually willing to reproduce its wording.

I would first remove exact duplicates using a stable hash over a carefully canonicalized representation. Canonicalization might collapse repeated whitespace and normalize line endings, but it must be conservative. Lowercasing is usually harmless for prose and potentially harmful for code, where UserID and userid may be different names.

Then I would find near duplicates. A common approach is to split documents into word or character shingles, compute compact signatures such as MinHash, and use locality-sensitive hashing to find candidates before calculating a more exact similarity. For code, I would add file-, function-, and repository-level comparisons. Forks are not automatically duplicates: a fork can carry a different license, meaningful changes, or important historical context.

Deduplication should happen across sources, not just within each source. Keep one canonical record, but retain all source and license metadata. If ten licensed copies of a document have different rights, deleting nine copies must not delete the rights evidence.

Quality filtering

Quality is not a single score. I would combine cheap deterministic checks with classifiers and human audits:

• language identification and minimum-content checks;
• boilerplate, navigation, SEO spam, and repeated-template detection;
• malware and secret scanning for code;
• document length and character-distribution checks;
• safety and sexually explicit content policies appropriate to the product;
• domain-specific checks for code, mathematics, documentation, and conversational text.

A perplexity filter can identify text that looks machine-generated, malformed, or unusually noisy, but it is not a truth detector. A fluent falsehood can pass. A dialect, minority language, or unconventional but valuable technical document can fail. I would therefore use filters to score, route, or downweight data before using them as irreversible deletion rules. Auditing false positives by language and domain matters more than boasting about one impressive aggregate “quality score.”

The first observable failure from poor filtering is often a strange training mixture: validation loss improves, but samples contain navigation menus, duplicated disclaimers, broken markup, or endless SEO paragraphs. For code, the symptom may be a model that emits plausible-looking snippets with missing imports and insecure copy-paste patterns.

Benchmark contamination

Benchmark contamination means training data contains evaluation questions, answers, or close variants. It can make a model look capable because it has seen the exam rather than learned the underlying skill.

I would maintain a registry of evaluation items, including exact text, answer choices, reference answers, known solutions, and useful paraphrase or translation variants. I would compare candidate training data using exact matching, n-gram or shingle overlap, and semantic review of high-similarity matches. Embeddings are useful for finding candidates, but not sufficient for a final decision: two legitimate explanations of the same theorem may be semantically close without being leaked test answers.

The registry should cover more than famous question strings. For code evaluations such as HumanEval, search for canonical solutions and distinctive function prompts. For exams such as MMLU or arithmetic sets such as GSM8K, search for copied questions, worked solutions, answer keys, and pages that reproduce the benchmark format.

A time cutoff helps, but it does not solve the problem. MMLU, GSM8K, and many other evaluations have been public, so a web crawl from before the model’s training run may still contain them. If a match cannot be confidently removed, I would mark the affected evaluation as contaminated and report it rather than quietly publishing the score. I would also keep private, newly authored holdouts that never enter the corpus pipeline.

Licensing and provenance

“Publicly accessible” does not mean “public domain,” and a permissive code license is not interchangeable with a proprietary data license. For each source, I would record the license or contractual basis, attribution and notice obligations, restrictions on redistribution or derivatives, geographic scope, expiration, and whether the source contains third-party material.

Web terms are often ambiguous. Robots exclusion rules can inform crawling policy, but they are not a universal substitute for a license. I would involve counsel for unresolved commercial or jurisdiction-specific questions rather than turning an engineering assumption into a legal conclusion.

The model-card and dataset report should identify source families and known limitations. A deletion request should be operational: remove the source from the next corpus build, invalidate derived shards and caches, and decide whether retraining or a validated unlearning procedure is required. Removing a row from a database does not make information disappear from weights already trained on it.

PII and secrets

Personally identifiable information, or PII, is information that can identify or single out a person. It includes obvious values such as email addresses, phone numbers, government identifiers, and precise addresses, but also combinations of ordinary facts that identify someone. Code and issue trackers add API keys, private URLs, credentials, and internal customer data.

I would use several detectors: regular expressions for structured identifiers, secret scanners for credentials, named-entity recognition, classifiers for contextual personal data, and source-specific rules. Regex alone misses “my home address is…” and produces false positives on version numbers or public support emails.

For high-risk material, remove it rather than merely replacing it with a stable token. A stable replacement can itself preserve linkage. Preserve only the metadata needed for auditing, with access controls and retention limits. Test the trained model for verbatim extraction using canaries and targeted prompts. Public PII is still PII; availability on the web does not remove the privacy concern.

Differential privacy versus utility

Differential privacy is a mathematical guarantee limiting how much the model’s output can change when one person’s data is added or removed from the training set. In language-model training, differentially private stochastic gradient methods typically clip each example’s gradient and add calibrated noise, then track a privacy budget expressed using parameters such as epsilon and delta.

The unit matters. Record-level privacy protects one record. User-level privacy protects all records belonging to one person. If one developer has contributed 1,000 code files, record-level protection may provide very little protection for that developer.

Stronger privacy generally means more clipping or noise, which can erase rare facts, long-tail language, and specialized code before it noticeably harms common patterns. The cost is especially painful for smaller datasets and full pretraining, where per-example gradient handling and privacy accounting are expensive. More data can recover some utility, but it does not make the trade-off disappear.

I would not apply DP as a ceremonial checkbox to an enormous public-web corpus. First minimize data, remove sensitive material, enforce access controls, and avoid training on data without a defensible rights basis. For genuinely sensitive user data, I would compare user-level DP pretraining or private fine-tuning against a non-private baseline, report the privacy parameters and attack results, and measure utility on both common and rare capabilities. A lower epsilon is stronger privacy, but no epsilon is meaningful without stating the neighboring-dataset definition, sampling assumptions, and accounting method.

Common mistake: differential privacy does not guarantee that a model will never output a person’s name, and PII scrubbing does not provide a formal privacy guarantee. They address different failure mechanisms.

The senior-level trade-off

I would ship a smaller, well-provenanced corpus over a larger corpus with unknown rights and uncontrolled repetition. But I would not blindly maximize cleanliness either. Removing every document that resembles a benchmark, every text containing a name, or every unusual code sample can destroy useful coverage.

The right output is a versioned corpus with measured exclusions, retained uncertainty, contamination reports, source-level rights decisions, PII-removal metrics, and utility comparisons. The model should be able to explain what it learned from, and the team should be able to explain what it deliberately did not use.

What they’ll ask next

How would you measure near-duplicate removal?
Use exact hashes for identical normalized records, MinHash or another shingling method to generate near-duplicate candidates, then validate thresholds on sampled pairs. Report both token reduction and cluster-size distributions; a sudden giant cluster often means the canonicalizer is too aggressive.

Would you remove every benchmark-related document?
No. I would remove or quarantine known test material and close solution matches where policy requires it, but ordinary educational material may remain if it does not reproduce evaluation content. I would report uncertain matches and rely on private, contamination-resistant holdouts.

When would you choose differential privacy?
When the threat model concerns membership or memorization of sensitive people’s data and the privacy guarantee is worth the measured utility and compute cost. For public data with unclear rights, DP is not a license to use it; governance and data minimization come first.

One line to say in the room:
“I’d treat the corpus as a versioned, auditable product: deduplicate and filter it, prove what we are allowed to use, quarantine evaluation content, remove sensitive data, and choose differential privacy only after measuring its privacy benefit against the capability we would lose.”