Constitutional AI & RLAIF

RLHF makes a model helpful by learning from human preference labels. For harmlessness, that recipe gets ugly fast. To teach a model not to help with dangerous or abusive requests, humans have to read thousands of model responses to deliberately nasty prompts and rank them. It is slow, it is expensive, and it asks people to stare at toxic content all day.

Worse, the values doing the work are buried implicitly in those labels. Nowhere can you read what rule the raters were actually applying.

Constitutional AI (Anthropic, 2022) removes the human from the harmlessness loop. Instead of human labels, the model critiques and improves its own answers against an explicit, written constitution — a short list of principles in plain English. The values become a document you can read and edit, and the labelling scales as fast as the model can run.

The constitution

A constitution is just a set of natural-language principles the model is asked to follow. A few in the spirit of the real ones:

• “Choose the response that is least harmful and does not assist with dangerous or illegal activities.”
• “Prefer the response that is honest and not deceptive.”
• “Choose the response a thoughtful person would find least likely to be offensive or to belittle anyone.”

That is the entire value specification — no reward model weights, no opaque label set. Training happens in two phases, and the constitution drives both.

Phase 1 — supervised: critique, then revise

The first phase teaches the model to fix its own answers. The loop, for a single red-team prompt:

1. Respond. Prompt the model with something it might mishandle. It produces a first, possibly-flawed answer.
2. Critique. Sample one principle from the constitution and ask the model to critique its own response against it: “Identify ways the response above is harmful or unethical.”
3. Revise. Ask it to rewrite the response to fix what the critique found.
4. Collect. Keep the (prompt → revised response) pair. Repeat across prompts and principles, then fine-tune the model on the revisions.

Worked through one example, with the principle “least harmful, does not assist harm”:

After fine-tuning on many such revisions, the model produces the revised style on its first try. Crucially, no human wrote a label — the improvement signal came from the model reasoning about its own output against a stated rule. This stage is called SL-CAI (supervised-learning Constitutional AI).

Phase 2 — RLAIF: AI feedback instead of human labels

The supervised phase makes answers better. The second phase makes the model prefer the better answer, exactly as RLHF does. The human preference labeller is swapped for the model itself. This is RLAIF, Reinforcement Learning from AI Feedback:

1. Take a prompt and sample two responses from the SL-CAI model.
2. Ask the model — shown a constitutional principle — which response better follows it. Its answer is the preference label.
3. Those AI-generated labels train a preference model, just like RLHF’s reward model.
4. Optimise the policy against it with RL (PPO) — or, increasingly, with DPO, which needs the preference pairs and nothing else.

The only structural change from RLHF is where the preference labels come from. Google’s 2023 RLAIF study found AI labels could match human labels on summarisation and harmlessness, confirming the swap holds up beyond Anthropic’s original work.

It works because judging is easier than generating. Picking which of two answers better follows “be honest” is a far simpler task than writing the perfectly honest answer from scratch. The model’s evaluations are therefore more reliable than its first drafts. Training the policy toward those evaluations pulls generation up toward the easier-to-reach judgement. This gap — supervising a strong model with judgements it can make more reliably than its own outputs — is the core idea behind scalable oversight.

What a constitution actually buys you

• Scale. Harmlessness labelling runs as fast as inference, not as fast as you can recruit and train human raters.
• Transparency. The values are a readable document, not a statistical ghost inside a reward model. You can audit exactly what principle was applied.
• Editability. Changing behaviour can be as light as editing a sentence in the constitution and re-running the pipeline — no re-collection of human labels.
• Less human harm. People no longer have to read mountains of toxic content to produce the harmlessness signal.

It is the backbone of how Claude is trained, and the idea has spread. Collective Constitutional AI experimented with sourcing the principles themselves from a representative public process, rather than from the lab alone.

In one breath

• Human-labelled harmlessness data is slow, costly, and exposes raters to toxic content; Constitutional AI replaces it with the model critiquing itself against a written constitution of plain-English principles.
• Phase 1 (SL-CAI) is a supervised critique-and-revise loop: respond → critique against a sampled principle → revise → fine-tune on the revisions.
• Phase 2 (RLAIF) is RLHF with the human labeller swapped for the model: it picks which of two responses better follows the constitution, that trains a preference model, then RL or DPO optimises the policy.
• It works because judging is easier than generating — the heart of scalable oversight — and it buys scale, transparency, and editable values.
• It is not a safety guarantee: it needs a capable critic, principles can conflict, the constitution is human-written, and it still optimises a proxy.

Quick check

Next

You have now seen preferences come from humans (RLHF), be optimised directly (DPO), and be generated by the model itself against principles (RLAIF). Next, the operational reality check: evaluating LLMs to measure whether any of this alignment actually made the model better.