Alignment: SFT, RLHF & DPO

Ask a raw, freshly pretrained model “What is the capital of France?” and you might get back: “What is the capital of Germany? What is the capital of Spain?” It isn’t broken. It is doing exactly what pretraining taught it — predicting the next most-likely text — and the most likely continuation of a quiz question is more quiz questions. A base model completes documents. It does not yet know that a question is a request meant to be answered.

So how does that text-completer become ChatGPT or Claude — something that can:

• answer
• help
• politely refuse

Not in pretraining. It happens afterward, in a stage called alignment, and it has three named steps worth understanding.

Why pretraining isn’t enough

Pretraining reads a large slice of the internet and learns two things superbly:

• knowledge (facts, patterns, code)
• fluency (grammatical, plausible text)

What it never learns is intent — that the human on the other side wants their instruction followed, wants the answer to be helpful, and wants it kept harmless. That behavior has to be trained in on purpose. Alignment is that training.

Stage 1 — SFT: show, don’t tell

The first step is SFT, Supervised Fine-Tuning (also called instruction tuning). It is fine-tuning — exactly the LoRA-or-full mechanics from the last lesson — on a dataset of demonstrations. The dataset contains thousands of (instruction → ideal response) pairs that humans wrote by hand. “Summarize this email → here is a good summary.” “Write a polite refusal → here is one.”

The model imitates the shape of these answers. After SFT it stops continuing the quiz and starts answering it. Think of SFT as an apprenticeship: the model watches a skilled human do the task right, many times, and learns to copy the form.

But imitation has a ceiling. SFT teaches one good answer per prompt. It can’t easily express “this answer is good, but that one is better.” A lot of what makes an assistant pleasant is exactly that kind of comparison:

• the right level of detail
• tone
• when to hedge

Stage 2 — RLHF: learn from preference

RLHF — Reinforcement Learning from Human Feedback — teaches preference instead of imitation. The recipe:

1. Collect comparisons. For a prompt, sample several responses from the SFT model and have humans rank them: A is better than B.
2. Train a reward model. A separate model learns to predict those human rankings — it outputs a scalar “how much would a human like this?” score.
3. Optimize with RL. Now fine-tune the assistant (typically with PPO) to produce responses the reward model scores highly. A KL-divergence penalty leashes it to the SFT model, so it can’t drift into gibberish that happens to fool the reward model.

That KL leash matters: without it, RL will find weird, high-scoring nonsense. RLHF works, and it powered the first wave of chat models. But it is a lot of moving parts: a reward model to train and maintain, plus a finicky RL loop that is sensitive to hyperparameters.

Stage 2, simpler — DPO

DPO — Direct Preference Optimization — reaches the same goal and throws away the reward model and the RL loop entirely. Its insight is that the preference data alone already tells you everything. You can convert “chosen beats rejected” directly into a loss on the model itself. It is a classification-style objective that raises the probability of the chosen response and lowers the rejected one. The loss is gently anchored to a frozen reference (the SFT model).

# The DPO objective, in code shape — this is the idea, not a training run.
# For one preference pair (prompt, chosen, rejected):
#   logp_*  = log-prob the CURRENT model assigns to a response
#   ref_*   = log-prob the FROZEN reference (the SFT model) assigns
# No sampling, no reward model: just push 'chosen' above 'rejected'.
import torch.nn.functional as F

def dpo_loss(logp_chosen, logp_rejected, ref_chosen, ref_rejected, beta=0.1):
    chosen_gap   = logp_chosen   - ref_chosen      # how much more the model now prefers 'chosen'
    rejected_gap = logp_rejected - ref_rejected    # ...vs 'rejected', relative to the reference
    return -F.logsigmoid(beta * (chosen_gap - rejected_gap))

No reward model, no PPO, far more stable to train — which is why DPO (and its relatives) became the common default for open models. The beta plays the role of the KL leash: small beta stays close to the reference, large beta lets the model move further to satisfy the preferences.

One preference pair, traced

Make it concrete. Take a single training example — a prompt and two correct answers, one preferred:

Both answers are factually fine. The difference is fit: only the chosen one obeys “to a 5-year-old.” Now watch each stage handle it:

• SFT trained on demonstrations might produce either style — it learned “give a correct explanation,” not “match the requested reading level.” It has no signal that one is better.
• Preference training does have that signal. RLHF would score the chosen answer higher via the reward model; DPO raises the chosen answer’s probability over the rejected one directly. Either way, the model learns to prefer the simple, on- target style for this kind of prompt — without anyone writing a rule for it.

That is the whole arc: pretrain for knowledge, SFT for the answer shape, preference training for the judgment of which answer is better.

The reward model is optimized for human approval, which is only a proxy for “helpful.” Optimize any proxy hard enough and the model learns to game it:

• padding length
• flattering the user
• sounding confident

This is reward hacking (a face of Goodhart’s law — when a measure becomes a target, it stops being a good measure), and the sycophancy it produces is one of alignment’s open problems.

In one breath

• A base model completes text; it doesn’t follow instructions until it is post-trained. That post-training is alignment.
• SFT (instruction tuning) fine-tunes on human (instruction → answer) demonstrations — the model imitates the shape of a good answer.
• RLHF adds preference: rank answers, train a reward model, optimize with PPO under a KL leash to the SFT model so it can’t drift.
• DPO reaches the same goal with no reward model and no RL loop — it turns preference pairs straight into a stable loss; now the common default.
• All preference training optimizes a proxy for “helpful,” so it can be gamed:
  • reward hacking
  • sycophancy
  • padding Alignment shapes behavior, not truth.

Quick check

Next

You now know how a base model becomes an assistant, and how to adapt one to your own behavior with LoRA. Next, the operational side: evaluating LLMs so you can tell whether any of this actually worked.