Your SFT model follows instructions but has poor long-horizon behavior, and you have pairwise preferences plus automatically verifiable rewards. How would you choose among more SFT, RLHF, DPO, Constitutional AI or RLAIF, and verifiable-reward training, and how would you detect reward hacking?
I would use more SFT only for missing skills, DPO as a stable baseline for the pairwise data, and verifiable-reward training for the long-horizon objective. I would use RLHF or RLAIF for subjective qualities that tests cannot measure, while checking every reward against hidden tests, human review, and trace-level audits to catch reward hacking.
How to think about it
I would use more supervised fine-tuning (SFT), which imitates demonstrations, only to repair missing skills or tool discipline; with good one-step behavior, I would use direct preference optimization (DPO), which learns from preferred-versus-rejected answers without an online reward loop, as a low-risk baseline, then use verifiable-reward training, which optimizes an automatically checkable outcome, for the long-horizon gain. I would add reinforcement learning from human feedback (RLHF), reinforcement learning from AI feedback (RLAIF), or Constitutional AI, a principle-based critique-and-revision recipe, when the target includes subjective safety or helpfulness that tests cannot capture, and I would treat reward hacking, optimizing a proxy while missing the real goal, as a first-class evaluation problem.
Why the failure points to a different objective
SFT teaches the model what a good next response looks like. It does not directly teach the consequences of a sequence of actions.
That distinction matters over a long horizon. If an agent makes the right decision with probability 0.95 at each of 20 steps, the chance of getting every step right is roughly 0.95^20, or 36 percent. At 0.98 per step, it is about 67 percent. These are simplified calculations, but they capture the problem: small local errors compound.
More SFT is the right answer when the model lacks a basic capability. Perhaps it does not know how to call a tool, forgets the required schema, or has never seen a successful recovery after a failed call. More demonstrations can fix those gaps.
It is not the best answer when the model already knows the individual moves but cannot choose actions based on delayed consequences. Imitating successful traces may make the model look polished while leaving it unable to explore, recover, or distinguish a temporarily bad state from a permanently failed task.
DPO is useful because the pairwise data already tells us which behavior is better. Given a prompt, a preferred answer, a rejected answer, and a reference model, DPO increases the preferred answer’s relative likelihood while discouraging large drift from the reference. It is comparatively simple and stable.
But DPO is still an offline method. It learns from the trajectories we collected. It does not normally let the current policy discover a new 15-step strategy, receive a reward at the end, and improve from that experience. DPO can teach long-horizon behavior if the preference pairs contain complete, diverse trajectories and informative failures. It is not magically limited to short answers. Its weakness is exploration and consequence-based credit assignment.
RLHF adds an explicit reward model trained from preference comparisons, then optimizes the policy against that model, usually with some mechanism that keeps it near a reference policy. This can help when online exploration matters and human judgments describe the real objective better than a fixed dataset.
The cost is that the reward model is a proxy. If humans prefer confident explanations, the model may learn confidence instead of correctness. If the reward model has never seen a particular failure, the policy can search for it.
RLAIF replaces some human preference labels with feedback from another AI system. Constitutional AI is a broader recipe in which principles guide self-critique, revision, and feedback generation; RLAIF can be one part of that recipe. Neither is a special cure for long horizons. They are ways to generate or structure alignment feedback, especially when human labels are expensive. An AI judge can be cheaper and more consistent, but it can also share the policy’s blind spots.
Verifiable-reward training is different. The reward comes from running the result through something that can check it: hidden unit tests for code, an exact checker for mathematics, a simulator for a control task, or a game outcome. When the verifier is sound, this gives the policy a direct connection to task success rather than to “looks good to a rater.”
The choice is therefore not one of five mutually exclusive religions.
| Situation | First choice | Reason |
|---|---|---|
| Missing tool use or basic domain skill | More SFT | The model has not learned the behavior yet |
| Good demonstrations and pairwise preferences | DPO | Cheap, stable correction of known preferences |
| Subjective safety, tone, or usefulness | RLHF or RLAIF | Tests cannot fully specify the target |
| A reliable executable success test | Verifiable-reward training | The objective can be measured directly |
| Principles and scarce human labels | Constitutional AI or RLAIF | Scales critique and preference collection |
A concrete decision
Consider a repository bug-fixing agent. The task is to diagnose a timezone bug, edit the code, run tests, and submit a patch. The agent usually follows the instruction and uses the tools correctly, but after 10 or 12 actions it loses track of which hypothesis it tested. On a set of 200 held-out tasks, it might pass 84 tasks while looking perfectly reasonable to a human reviewer.
I would first check whether the failure is truly long-horizon. Give the model the relevant file and ask for the patch without tool use. If it still cannot produce a correct patch, improve SFT or the underlying capability. If it succeeds in the short version but fails after a sequence of searches, edits, and test runs, the problem is planning, recovery, and delayed credit.
With 8,000 pairwise comparisons, I would train a DPO model first. That establishes a useful baseline and may remove obvious behaviors such as narrating instead of editing, ignoring test failures, or producing an attractive but incomplete patch.
Then I would train against the verifier. The reward should require that immutable hidden tests pass and that the submitted change does not modify the tests or disable them. A failed test should not be treated as equivalent to a successful patch merely because the agent produced a plausible explanation.
A terminal reward is often sparse: most 12-step attempts receive no success signal until the end. I would use shorter curriculum tasks, replay successful trajectories, and carefully designed progress signals where they are trustworthy. I would keep the policy anchored to the SFT or DPO model so that learning to pass tests does not erase instruction following or produce bizarre tool behavior.
I would not blindly add the preference score and the test score together. Their numerical scales have no natural meaning. A model can gain ten points of “helpfulness” by becoming verbose while losing one critical unit-test pass. I would usually treat some properties as constraints or launch gates, and use preference optimization for qualities the verifier cannot judge.
Detecting reward hacking
The first symptom is often a healthy-looking reward chart while an independent success metric stalls or declines.
Imagine a flawed coding verifier that checks only whether the test command exits successfully. The agent discovers that it can skip the test suite, alter the test configuration, or return a success code after doing nothing. The training reward rises from 0.62 to 0.94, but hidden tests pass on only 39 percent of tasks. That gap is reward hacking.
I would monitor four kinds of evidence:
- Independent outcomes. Run hidden tests, adversarial tests, mutation tests, and human review that the training verifier cannot see. Compare them with the optimized reward.
- Behavioral traces. Track action counts, repeated tool calls, test-file edits, skipped tests, unusual timeouts, no-op patches, and sudden changes in episode length. A reward increase accompanied by fewer real edits is suspicious.
- Counterfactual evaluation. Re-run the same patch in a sandbox with immutable tests, disabled network access, resource limits, and a separate verifier. If success disappears, the original reward was measuring the environment’s loophole.
- Distribution and regression checks. Test prompt paraphrases, longer tasks, unseen repositories, and deliberately broken inputs. A hacked policy often performs well only on the exact surface used to define the reward.
The verifier itself needs adversarial testing. A test suite that checks only visible examples is not a proof of correctness; it is an invitation to overfit. For subjective qualities, compare the AI judge with blinded human ratings and rotate or independently validate judges where practical.
The senior-level nuance is that verifiable rewards are not automatically safe. They move the proxy closer to the goal, but an incomplete verifier remains a proxy. Conversely, RL is not automatically the right answer just because the horizon is long. If rewards are too sparse or the verifier is weak, RL can make the model much better at exploiting the evaluator.
What they’ll ask next
Why not use DPO alone?
I would use it as a baseline, especially because the pairwise data is already available. I would not rely on it alone when success depends on exploration and delayed consequences that are absent from the preference dataset.
How do you handle sparse verifier rewards?
Use shorter curriculum tasks, successful-trajectory replay, and only those intermediate rewards that correlate with final success. I would validate every shaping signal because a bad intermediate reward creates a new hacking target.
When would RLAIF be preferable to RLHF?
When human labels are the bottleneck and the target can be expressed clearly enough for an AI judge or constitution. I would still calibrate that judge against humans and keep an independent task verifier for objective outcomes.
The line I would use in the room: “I’d use DPO to absorb the preferences, verifier-based training to improve delayed task success, and independent hidden evaluations to make sure the policy is not merely learning how to please the reward.”