Constitutional AI and RLAIF

Bai et al. (arXiv:2212.08073, 2022) asked: what if we replaced the human labeler with an AI that reads a list of principles? Constitutional AI has two phases — self-critique and revision under a constitution, then RL from AI Feedback. The technique coined the term RLAIF and shipped in the Claude 1 post-training pipeline. On 21 January 2026 Anthropic published a rewritten Claude constitution: explanatory reasoning over prescriptive rules, a four-tier priority hierarchy, and the first major-lab formal acknowledgment of uncertainty about model moral status. Released under CC0 1.0.

Learn 1 hr Python (stdlib, toy self-critique-and-revise loop)

Prerequisites: Phase 18 · 01 (InstructGPT), Phase 18 · 02 (Reward hacking)

#Learning Objectives

Describe the two phases of Constitutional AI (critique-and-revise SFT, RL from AI feedback) and the role of the constitution in each.
Explain why replacing a human preference labeler with an AI labeler is not a "cheaper" RLHF — it changes which failure modes the pipeline has.
Summarize the four-tier priority structure of the 2026 Claude constitution and what changed from the 2023 rewrite.
Describe Constitutional Classifiers and the drop from 23.7% compute overhead (v1) to ~1% (v2 / 2026).

#Learning Objectives

Keep going. The rest of this lesson is one step away.