Direct Preference Optimization (DPO)

RLHF works, but look at everything it needs: collect human rankings, train a separate reward model to imitate them, then run a PPO loop that samples from the policy, scores those samples, and nudges the weights — all while a KL penalty keeps the policy from drifting into nonsense. Three models in play, a reinforcement-learning loop that is famously twitchy to tune. The preference data is just pairs of answers, one marked better. Surely that is enough on its own?

Direct Preference Optimization says yes. It proves — with a short, exact piece of algebra — that the reward model and the RL loop can be folded into a single supervised loss on the policy. Same objective as RLHF, none of the machinery. This lesson derives that loss from scratch.

The starting point: preferences as probabilities

Every preference method begins with the same data: a prompt x, a chosen answer y_w (the winner), and a rejected answer y_l (the loser). To learn from it we need a model of how a hidden “quality” score turns into a choice.

The standard one is the Bradley-Terry model. If answer y has a real-valued reward r(x, y), the probability a rater prefers the winner over the loser is a sigmoid of the reward gap:

P(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) )

A bigger reward gap means a more confident preference; an equal gap means a coin flip (σ(0) = 0.5). Classic RLHF trains a reward model r to maximise the likelihood of the observed preferences under exactly this equation, then optimises the policy against that r. DPO’s whole trick is to skip the middle model.

The hinge: RLHF already has a closed-form answer

Here is the fact DPO exploits. The RLHF optimisation — maximise reward while staying close (in KL) to the reference policy π_ref (the SFT model) — is not some intractable thing PPO heroically approximates. It has a known exact solution:

π*(y | x)  =  (1 / Z(x)) · π_ref(y | x) · exp( r(x, y) / β )

The optimal aligned policy is just the reference policy re-weighted by the exponentiated reward. β is the KL strength (small β = move freely, large β = hug the reference), and Z(x) is a normaliser (the partition function) that makes the probabilities sum to 1. We never actually use this to compute π* — Z(x) sums over every possible response and is hopelessly expensive. We use it to do algebra.

The trick: solve for the reward

Take logs of that optimum and rearrange to put the reward on one side:

r(x, y)  =  β · log( π*(y | x) / π_ref(y | x) )  +  β · log Z(x)

Read that slowly: the reward is recoverable from the policy itself — it is β times the log-ratio of the aligned policy to the reference, plus a term that depends only on x, not on y. We never needed a separate reward network; the policy implies its own reward.

Now substitute this expression for r back into the Bradley-Terry equation. We take a difference of two rewards (winner minus loser), and the awkward β · log Z(x) term is identical for both answers — so it cancels:

P(y_w ≻ y_l | x) = σ( β·log[π(y_w|x)/π_ref(y_w|x)]  −  β·log[π(y_l|x)/π_ref(y_l|x)] )

Z(x) is gone. The thing that made the optimum incomputable has vanished, and what remains is written entirely in quantities we can evaluate — log-probs the policy and the frozen reference assign to two given answers.

The DPO loss

Maximising the likelihood of the preference data under that last equation is the DPO loss. For one pair it is:

L_DPO = − log σ( β·log[π_θ(y_w|x)/π_ref(y_w|x)]  −  β·log[π_θ(y_l|x)/π_ref(y_l|x)] )

It is an ordinary classification loss — push the log-ratio of the winner above the log-ratio of the loser — trained by plain gradient descent on the policy π_θ. No sampling, no reward model, no PPO. That quantity in brackets has a name: the implicit reward.

r̂(x, y) = β · log( π_θ(y | x) / π_ref(y | x) )

DPO never builds a reward model because every policy is one: r̂ is the reward DPO is implicitly fitting, read straight off the weights.

Watch the loss cross the line

The implicit-reward margin r̂(y_w) − r̂(y_l) is the whole story. When it is positive the model prefers the winner (good, low loss); when negative it still prefers the loser (bad, high loss). The dividing line is the loss at margin zero, −log σ(0) = log 2 ≈ 0.69:

Trace one pair through two training snapshots. The reference is frozen, so only the policy’s log-probs move:

import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
beta = 0.1

def dpo(logp_w, logp_l, ref_w, ref_l):
    r_w = beta * (logp_w - ref_w)     # implicit reward of the chosen answer
    r_l = beta * (logp_l - ref_l)     # implicit reward of the rejected answer
    margin = r_w - r_l
    loss = -np.log(sigmoid(margin))
    return np.round([r_w, r_l, margin, loss], 4)

# A: model still assigns the rejected answer the higher implicit reward
print("A:", dpo(logp_w=-5.0, logp_l=-3.8, ref_w=-5.0, ref_l=-5.0))
# B: after gradient steps, the chosen answer now wins
print("B:", dpo(logp_w=-4.2, logp_l=-5.0, ref_w=-5.0, ref_l=-5.0))
print("neutral loss -log(0.5):", round(float(-np.log(0.5)), 4))

A: [ 0.     0.12  -0.12   0.7549]
B: [ 0.08   0.     0.08   0.6539]
neutral loss -log(0.5): 0.6931

In snapshot A the model gives the rejected answer the higher implicit reward (0.12 vs 0.0), the margin is negative, and the loss sits above the ln 2 line — the gradient will push the chosen answer’s log-prob up. By snapshot B the chosen answer leads (0.08 vs 0.0) and the loss has dropped below ln 2. That is all training is: drag every pair across the line.

What the gradient actually does

The DPO gradient has a clean shape. For each pair it scales the update by σ(r̂(y_l) − r̂(y_w)) — a weight that is large when the model is most wrong (it currently prefers the loser) and near zero once the winner is comfortably ahead. So DPO automatically spends its effort on the pairs it is still getting backwards and stops fussing over ones it already ranks correctly. It is a soft, self-pacing version of “fix your worst mistakes first.”

The β knob, and where DPO bites back

β is the same KL strength as in RLHF. Small β (say 0.1) lets the policy move far from the reference to satisfy preferences — more alignment, more risk of degenerate behaviour. Large β keeps it tethered — safer, but it learns the preferences less aggressively. There is no PPO to tune, but β still trades the same things.

The family it started

DPO’s “preferences as one loss” framing spawned a whole family, each patching a weakness:

The core idea is unchanged in all of them: a preference is a gradient signal you can apply directly to the policy.

In one breath

• RLHF needs a reward model plus a PPO loop; DPO collapses both into one supervised loss on the policy, from the very same preference pairs.
• It starts from Bradley-Terry (P(y_w ≻ y_l) = σ(reward gap)) and the fact that the KL-constrained RLHF objective has a closed-form optimum: π* ∝ π_ref · exp(r/β).
• Solving that for the reward gives r = β·log(π/π_ref) + β·log Z; substituting back into Bradley-Terry makes the intractable Z(x) cancel, leaving a loss in policy log-ratios only.
• The bracketed term is the implicit reward r̂ = β·log(π_θ/π_ref); training just pushes each pair’s margin r̂(chosen) − r̂(rejected) positive — past the ln 2 ≈ 0.69 neutral loss.
• Simpler than RLHF but not free: watch for length bias and over-fitting, tune β like the old KL leash, and reach for IPO/KTO/ORPO/SimPO when DPO strains.

Quick check

Next

DPO assumes you already have preference pairs — usually labelled by humans. The next lesson, Constitutional AI, shows how to generate the preference signal from the model itself against a written set of principles, so harmlessness training no longer needs a human to read every toxic sample.