Differential privacy for LLMs

Large models memorize. Trained on text that includes private emails, API keys, medical notes, and copyrighted books, a model can be prompted to regurgitate chunks of it verbatim — extraction attacks have pulled real names, phone numbers, and addresses out of production models. Two related risks: membership inference (figure out whether a specific record was in the training set) and data extraction (recover the record itself). Differential privacy (DP) is the rigorous defense — not a filter, but a mathematical guarantee.

The guarantee: epsilon bounds one person’s influence

DP makes a precise promise: the output of a computation is almost the same whether or not any one individual’s record is included. Formally, for two datasets differing by a single record, the probability of any outcome changes by at most a factor of exp(epsilon). That epsilon (the privacy budget) is the knob: small epsilon = strong privacy (the individual is well hidden) but more noise; large epsilon = weak privacy but a more accurate result.

The basic tool is the Laplace mechanism: to release a numeric query privately, add random noise scaled to sensitivity / epsilon, where sensitivity is how much one record can change the answer. A count has sensitivity 1 (adding one person changes it by 1):

import numpy as np
rng = np.random.default_rng(0)

true_count, sensitivity = 1000, 1
print("epsilon  scale_b  noisy_answer  (true=1000)")
for eps in [10.0, 1.0, 0.1]:
    b = sensitivity / eps                 # Laplace scale: smaller epsilon -> more noise
    noisy = true_count + rng.laplace(0.0, b)
    print(f"{eps:>6}  {b:>6.1f}  {noisy:>11.1f}")

epsilon  scale_b  noisy_answer  (true=1000)
  10.0     0.1       1000.0
   1.0     1.0        999.4
   0.1    10.0        975.0

At epsilon=10 the noise is tiny — the answer is essentially exact, but it barely hides anyone. At epsilon=0.1 the noise (±10ish) masks whether any single person was counted, at the cost of an answer that’s only roughly right. That dial — privacy bought with accuracy — is differential privacy.

Training privately: DP-SGD

To make the trained model itself private, you privatize the gradients. DP-SGD changes ordinary training in two ways each step: clip every per-example gradient to a maximum norm (so no single example can dominate the update), then add Gaussian noise to the summed gradients before applying them. No individual example moves the weights much, so the model can’t memorize any one of them.

The price is real. The noise lowers accuracy and training is slower; and crucially, the privacy cost accumulates over steps (composition) — each access to the data spends from the epsilon budget, so you track the total spent across the whole training run.

In one breath

• LLMs memorize training data and can be made to leak it — membership inference (was this record used?) and extraction (recover the record, e.g. PII).
• Differential privacy is a guarantee: including or excluding any one record changes the output probability by at most exp(epsilon) — so the output reveals almost nothing about any individual.
• Epsilon is the privacy budget: small ε = strong privacy + more noise; large ε = weak privacy + more accuracy. The Laplace mechanism adds noise scaled to sensitivity / epsilon (the demo: ε=0.1 masks the count, ε=10 barely hides it).
• DP-SGD trains privately by clipping per-example gradients and adding Gaussian noise; the privacy cost composes across steps, spending the budget.
• DP costs utility and that cost is uneven — it hurts rare/minority patterns most — so it’s applied selectively and alongside dedup and PII scrubbing.

Quick check

Next

DP attacks the memorization that starts in the pretraining data; the noise it relies on is the Laplace/Gaussian distributions; and its uneven cost connects to broader questions of fairness in ML.