Differential privacy for LLMs

At 3 a.m., a hospital’s support chatbot starts completing a sentence with a patient’s full name, phone number, and diagnosis. The data team removed obvious names before training. The leak is still real.

A model does not store a tidy table called patient_records. It stores weights adjusted by millions of training examples. A rare sentence, an unusual API key, or a duplicated medical note can leave a stronger trace than an ordinary sentence. With the right prompts, an attacker may recover that trace.

This is the problem differential privacy, usually shortened to DP, is designed to address. DP does not promise that a model knows nothing. It promises something narrower and more useful: changing one protected record should barely change the distribution of what the released system produces.

Suppose our hospital has 100,000 discharge summaries. Alice’s summary is one of them. A private training procedure should produce nearly the same distribution over released models whether Alice’s summary is included or removed. DP bounds how much evidence the released output can provide about Alice’s presence; it does not make every membership test fail.

That is a guarantee about influence, not a slogan about secrecy.

Why language models memorize

An LLM predicts the next token: a word, word fragment, or punctuation mark. If its training text contains:

Patient ID 48192. Contact Alice Smith at 555-0147 about the biopsy.

training repeatedly pushes the model to make that exact continuation likely after a similar prefix.

Common text is learned as a broad pattern. A rare string can be learned specifically, especially when it is:

• long and distinctive;
• duplicated in the corpus; or
• trained repeatedly on a small or sensitive dataset.

Memorization means retaining a particular sequence more specifically than general language ability requires. Attackers can send many prompts and search for unusually confident, low-entropy completions. Extraction attacks have recovered names, phone numbers, and other training strings from language models.

Two risks are related but different:

• Membership inference: “Was this record used for training?”
• Data extraction: “Can I recover the record or part of it?”

Membership can be damaging even without extraction. Confirming that someone appeared in a clinic’s dataset may reveal that they received care.

Removing identifiers, deduplicating data, access controls, encryption, and output filters all help. None, by themselves, mathematically limit how much one remaining row can change learned parameters. DP changes the training rule so that influence is bounded.

The guarantee: compare neighboring datasets

Let:

• D contain Alice’s record.
• D' be identical except that Alice’s record is absent.

These are neighboring datasets: they differ in one protected unit. That unit must be chosen deliberately. It might be a document, customer, patient, or household.

A randomized algorithm, called a mechanism and written as M, satisfies epsilon-differential privacy when, for every possible set of outputs S:

P[M(D) in S] <= exp(epsilon) * P[M(D') in S]

The same condition must hold with D and D' swapped. No output event becomes much more likely merely because Alice was present.

epsilon is the privacy budget. It controls the maximum multiplicative change between output distributions:

• At epsilon = 0.1, exp(epsilon) is about 1.105.
• At epsilon = 1, the factor is about 2.718.
• At epsilon = 10, the factor is about 22,026, a much weaker guarantee.

The bound covers every observation an attacker could make, including prompts to a released private model, as long as those observations are only post-processing. Post-processing cannot worsen the formal DP guarantee.

Practical DP-SGD usually reports (epsilon, delta)-differential privacy:

P[M(D) in S] <= exp(epsilon) * P[M(D') in S] + delta

delta is a small additive failure allowance. Gaussian noise naturally gives this approximate guarantee. A report such as “epsilon equals 4” is incomplete without delta, the neighboring-unit definition, the sampling scheme, and the number of steps.

A common starting convention is to choose delta substantially below 1/N, where N is the number of protected units. For 100,000 patients, 1/N is 0.00001. This is a convention, not a law; delta should follow the threat model and privacy policy, not a library default.

The diagram below is a schematic, non-formal picture of pure DP, where delta is zero. The formal guarantee concerns the probability of every event S, not the distance between peaks.

A numeric mechanism: the Laplace mechanism

Before neural networks, consider a count. It exposes the central arithmetic.

Suppose a table has one row per visit and we release the total number of visits. Under add-or-remove adjacency for one visit, the sensitivity is 1: adding or removing one row changes the count by at most one.

The Laplace mechanism adds noise from a Laplace distribution centred at zero. Its scale is:

b = sensitivity / epsilon

Smaller epsilon means larger noise and less precision. The Laplace distribution has standard deviation sqrt(2) * b; its central 95 percent interval is approximately plus or minus 3b.

import math

true_count = 1000
sensitivity = 1

for epsilon in (10.0, 1.0, 0.1):
    scale = sensitivity / epsilon
    error_95 = scale * math.log(20)
    print(
        f"epsilon={epsilon:>4.1f}  "
        f"scale={scale:>5.2f}  "
        f"95pct_abs_error=+/-{error_95:>5.2f}"
    )

epsilon=10.0  scale= 0.10  95pct_abs_error=+/- 0.30
epsilon= 1.0  scale= 1.00  95pct_abs_error=+/- 3.00
epsilon= 0.1  scale=10.00  95pct_abs_error=+/-29.96

With epsilon 0.1, a count of 1,000 is typically within about 30 of the truth, while one visit’s contribution of 1 is difficult to distinguish from the noise. At epsilon 10, the count is more precise but the guarantee is weak.

The mechanism is not “add a random number and hope.” Noise is calibrated to sensitivity. If one record could change the query by 500, scale 1 / epsilon would not be private.

DP-SGD: bound each example, then add noise

LLM training uses gradients, whose influence can be unbounded. DP-SGD adapts stochastic gradient descent in two steps:

1. Compute a gradient for each training example.
2. Clip each gradient to a maximum norm, then add Gaussian noise to the aggregate before updating the weights.

Let g_i be one example’s gradient and C the clipping norm:

g'_i = g_i * min(1, C / norm(g_i))

A gradient shorter than C is unchanged. A longer gradient is rescaled to length C. Every example therefore has a bounded maximum contribution to the batch update.

Clipping alone is not privacy: an attacker could inspect the exact bounded sum. DP-SGD adds Gaussian noise. For L2 clipping under add/remove adjacency, the summed-gradient sensitivity is C, so the Gaussian standard deviation is commonly sigma * C, where sigma is the noise multiplier. Under replacement adjacency, sensitivity can be 2C, unless the accountant’s convention absorbs that factor.

If the implementation averages the batch, both the clipped sum and noise are divided by batch size; the accountant handles that scaling. The causal chain is:

1. clipping bounds sensitivity;
2. calibrated noise masks each example’s bounded contribution;
3. composition tracks the privacy cost over training.

Privacy cost accumulates because training touches the data repeatedly. This is composition. More steps, more epochs, larger sampling rates, and less noise generally increase final epsilon. Randomly sampling a small batch from a large dataset provides privacy amplification by subsampling, so the total is not usually a simple sum of per-step epsilons. Modern privacy accountants model these effects.

What the accountant must know

A credible DP-SGD report includes:

• protected unit and adjacency rule;
• dataset size and sampling rate;
• batch size and optimizer steps;
• clipping norm and noise multiplier;
• final epsilon at a stated delta; and
• every other data-dependent release or tuning decision.

Accountants often use Rényi differential privacy, or RDP, because it composes more tightly than basic bounds, then convert the result to (epsilon, delta). Do not confuse the noise multiplier with epsilon: the same multiplier can produce different epsilon values with different dataset sizes, sampling rates, or training lengths.

Hyperparameter search also spends privacy. Training ten private models, comparing private validation scores, and releasing the winner is a data-dependent process. Use public or synthetic tuning data where possible, or account for private evaluations and releases.

Production pattern

Define the protected person before choosing an algorithm. If each patient contributes one discharge summary, document-level DP may be close to patient-level DP. If a patient contributes 30 progress notes, protecting one note does not protect the patient. For user-level DP, aggregate or clip all of a user’s contributions together. This usually costs more utility because an entire history must fit under the clipping bound.

Reduce avoidable memorization too: deduplicate documents, remove secrets and identifiers, limit repeated boilerplate, and keep private evaluation data out of training. DP complements these controls; it does not replace them. Copyright risk is also separate from individual privacy.

For private fine-tuning:

1. Define the threat model and protected unit.
2. Establish a non-private baseline.
3. Choose clipping and noise ranges.
4. Run the accountant before training each configuration.
5. Evaluate quality, subgroup performance, memorization, and extraction resistance.
6. Record final (epsilon, delta) and its assumptions.
7. Audit adapters, evaluations, retrieval, logs, and exports for later non-private releases.

Parameter-efficient fine-tuning does not automatically provide privacy. A LoRA adapter is still calculated from sensitive examples and is a data-dependent release.

Failure modes

A private model can still produce a phrase found in the corpus. DP limits one protected unit’s influence; it does not erase common population-level facts.

When not to use DP

Do not add DP merely because a checklist says “privacy.” Ask what must be protected and what output must be released.

If data stays in a tightly controlled environment, access control and governance may address the immediate risk more directly. Exact counts for tiny groups may become unusable after noise. Strong DP can also reduce rare-subgroup signal, compute efficiency, and iteration speed. If the organisation cannot define its protected unit or track subsequent releases, an epsilon report creates false confidence.

DP is worth the cost when a valuable release must remain safe against attackers with auxiliary information: for example, public statistics, cross-organisation analytics, or models trained on data whose subjects cannot inspect every learned trace.

In one breath

• DP bounds how much one defined protected unit can influence a released model or statistic.
• Laplace noise scales with sensitivity divided by epsilon; DP-SGD clips per-example gradients, adds Gaussian noise, and accounts for composition.
• The guarantee depends on the protected unit, adjacency, delta, sampling, steps, and every later data-dependent release.
• DP limits individual influence; it does not scrub identifiers, erase common facts, encrypt data, or solve copyright by itself.

Quick check

Next

DP attacks memorization that begins in the pretraining data. The noise distributions behind its basic mechanisms are covered in distributions. Its uneven utility cost connects to fairness in machine learning.