Differential attention

Self-attention is supposed to focus — to pull in the few tokens that matter and ignore the rest. But softmax has a quiet flaw: it never outputs exactly zero. Every token gets at least a sliver of weight, and across a long context those slivers add up. In a 50,000-token window, the genuinely relevant tokens might collectively receive less attention than the vast sea of irrelevant ones, simply because there are so many of them. This is attention noise, and it is a measurable drag on long-context quality.

Differential attention (the DIFF Transformer, 2024) removes it with a trick borrowed from electrical engineering and noise-canceling headphones: if two signals carry the same noise, subtracting one from the other cancels the noise and leaves the difference — the signal.

The noise-canceling idea

A differential amplifier (and a noise-canceling headphone) takes two inputs that share a common background noise, and outputs their difference. The shared noise appears in both, so it subtracts away to zero; whatever differs between them survives. Differential attention applies exactly this to attention maps.

Instead of one softmax map, compute two, and output their weighted difference:

DiffAttn(X) = ( softmax(Q1 K1ᵀ / √d)  −  λ · softmax(Q2 K2ᵀ / √d) ) V

Both maps are trained to learn the same background attention pattern — the noise — but only the first also locks onto the relevant tokens. Subtract the second (scaled by a learnable scalar λ) and the common-mode noise cancels, leaving sharp weight on what matters. Watch it happen on one query row over six keys, where only key 1 is relevant:

import numpy as np

def softmax(z):
    z = z - z.max(); e = np.exp(z); return e / e.sum()

# Map 1 carries the signal (key 1) on top of a flat noise floor;
# Map 2 has only the same noise floor.
s1 = np.array([0.5, 3.0, 0.5, 0.5, 0.5, 0.5])
s2 = np.array([0.5, 0.5, 0.5, 0.5, 0.5, 0.5])
a1, a2 = softmax(s1), softmax(s2)
lam = 0.5
diff = a1 - lam * a2                      # the differential attention weights

irrelevant = [0, 2, 3, 4, 5]              # every key except the relevant key 1
print("standard softmax attn:", np.round(a1, 3))
print("  weight wasted on 5 irrelevant keys:", round(a1[irrelevant].sum(), 3))
print("differential attn    :", np.round(diff, 3))
print("  weight on those keys now         :", round(diff[irrelevant].sum(), 3))

standard softmax attn: [0.058 0.709 0.058 0.058 0.058 0.058]
  weight wasted on 5 irrelevant keys: 0.291
differential attn    : [-0.025  0.626 -0.025 -0.025 -0.025 -0.025]
  weight on those keys now         : -0.126

Standard softmax leaks 29% of its attention onto the five irrelevant keys. The differential map drives that common-mode noise to essentially zero (slightly negative — over-cancelled, which is fine), while the relevant key keeps the lion’s share. Same idea, drawn:

How it’s wired (and why it’s nearly free)

You do not double the compute. Each head’s query and key projections are split into two halves — Q1, K1 and Q2, K2 — so two smaller attention maps are computed in place of one full-size one. The FLOPs and parameters stay close to a normal multi-head layer; the only real addition is the learnable scalar λ, which is re-parameterised (as a difference of exponentials of learned vectors) so training stays stable and λ lands in a sensible range. It drops into the standard transformer block in place of ordinary attention.

Why the sharpness pays off

Cancelling attention noise has three downstream wins, all reported for the DIFF Transformer:

• Long-context retrieval. With the noise floor gone, a “needle” buried in a huge haystack keeps a clear share of attention instead of being drowned by thousands of irrelevant tokens — markedly better key-information retrieval.
• Fewer hallucinations. Many hallucinations come from the model attending to, and being swayed by, irrelevant context. Less noise means the output is anchored more firmly to the tokens that actually matter.
• Easier quantization. Sharper, cleaner attention produces fewer extreme activation outliers — the values that make low-bit quantization hard. So a diff-attention model is friendlier to compress for serving.

In one breath

• Softmax attention never outputs exactly zero, so it leaks weight onto every token — attention noise that buries the signal across a long context (29% wasted on irrelevant keys in the demo).
• Differential attention cancels it like a noise-canceling headphone: compute two softmax maps and output their weighted difference (softmax₁ − λ·softmax₂)V; the shared noise subtracts away, the signal remains.
• It is nearly free: each head’s Q/K are split into two halves (not doubled), adding only a learnable, re-parameterised scalar λ.
• The sharper pattern improves long-context retrieval (needle-in-haystack), reduces hallucination (less sway from irrelevant context), and eases quantization (fewer activation outliers).

Quick check

Next

Differential attention sharpens what the model attends to; sparse & sub-quadratic attention changes how much it computes, and FlashAttention changes how the same attention touches memory. Together they are the modern attention toolkit.