AI Engineering/Phase 10/Lesson 09

Constitutional AI and Self-Improvement

RLHF needs humans in the loop. Constitutional AI replaces most of them with the model itself. Write a list of principles, have the model critique its own outputs against those principles, and train on the critiques. DeepSeek-R1 pushed this further in 2025: let the model generate millions of reasoning traces, grade them with a rule, and run GRPO on the outcome. Most of the "alignment work" in a 2026 frontier model is the model alignment itself. This lesson builds both loops.

Build 45 min Python (stdlib + numpy)

Prerequisites: Phase 10, Lessons 06-08 (SFT, RLHF, DPO)

#Learning Objectives

Implement the Constitutional AI two-stage loop: self-critique plus self-revision, then preference training on the revised pairs
Derive the GRPO objective (DeepSeek-R1's group-relative policy optimization) and contrast it with PPO's value-function baseline
Generate verifiable reasoning traces with rule-based outcome rewards and score them without a separate reward model
Decide when self-improvement beats human preference data and when it collapses into mode seeking

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.