datarekha
LLMs June 8, 2026

RLHF is being replaced: how DPO teaches models what good means

RLHF aligned chat models with a reward model and a fragile RL loop. DPO drops both, learning the same preferences directly from chosen-vs-rejected pairs.

8 min read · by datarekha · llmalignmentrlhfdpopreference-optimization

The thing that turned raw language models into useful assistants was not more data or more parameters. It was alignment — teaching the model to prefer answers humans actually like over answers that are merely plausible. The original recipe for that, RLHF, worked brilliantly and was a genuine pain to run. The 2026 default, DPO, gets to the same place with a fraction of the machinery.

What RLHF actually does

Reinforcement Learning from Human Feedback is a three-act process:

  1. Supervised fine-tuning — teach a base model to follow instructions from example demonstrations.
  2. Train a reward model — collect human comparisons (“answer A is better than answer B”) and train a separate model to predict that human preference as a score.
  3. Reinforcement learning — optimize the language model to maximize that reward score, usually with PPO, while a reference model keeps it from drifting too far from sensible language.

It works, but look at how much is moving: three models in play, a reward model that can be gamed, and an RL loop that is notoriously sensitive to hyperparameters and prone to instability. Getting it right was real expertise.

What DPO realized

Direct Preference Optimization started from a clever observation: a language model is, mathematically, secretly its own reward model. If that is true, you do not need to train a separate reward model or run reinforcement learning at all. You can rearrange the math so the model learns directly from the same preference pairs — just make the chosen answer more likely and the rejected answer less likely, relative to a frozen reference model, with one straightforward classification-style loss.

No reward model. No PPO. No RL loop to babysit. Same goal — match human preferences — reached with ordinary supervised-style training.

RLHF3 models, unstable RLReward modeltrained on prefsRL loop (PPO)policy + reward + referencealigned modelDPO1 step, stablepreference pairschosen vs rejectedaligned modelone direct loss

Why simpler won

The appeal is almost entirely operational. DPO removes the most failure-prone parts of the pipeline — the separately-trained reward model that could be exploited, and the reinforcement-learning loop that demanded careful tuning. What is left is stable, cheap, and reproducible enough that a small team can align a model without an RL specialist on staff. When two methods reach a similar destination and one of them deletes an entire category of things that can go wrong, the field tends to vote with its feet.

Why this matters

Alignment used to be a moat: doing RLHF well took rare expertise and serious infrastructure. DPO lowered that bar enormously, which is a big part of why high-quality open models proliferated — tuning a model to be genuinely helpful stopped requiring a reinforcement-learning research team. If you ever fine-tune a model to match a particular tone, policy, or set of preferences, DPO is very likely the tool you will reach for, and it sits right alongside the supervised fine-tuning and adapter techniques that make customizing a model practical.

Skip to content