AI Engineering/Phase 18/Lesson 03

The Direct Preference Optimization Family

Rafailov et al. (2023) showed RLHF's optimum has a closed form in terms of the preference data, so you can skip the explicit reward model and optimize the policy directly. That insight spawned a family — IPO, KTO, SimPO, ORPO, BPO — each fixing a failure mode of DPO. In 2026, direct alignment algorithms ship more frontier post-training runs than PPO. But the over-optimization curve from Lesson 2 still applies: DAAs do not escape Goodhart, they just move where it bites.

Learn 1.3 hr Python (stdlib, six-variant preference-loss comparator)

Prerequisites: Phase 18 · 01 (InstructGPT), Phase 18 · 02 (Reward hacking), Phase 10 · 08 (DPO basics)

#Learning Objectives

Derive the DPO closed form from the RLHF-with-KL optimum.
State the failure mode each of IPO, KTO, SimPO, ORPO, BPO fixes in DPO.
Distinguish "implicit reward gap" from "preference strength" and explain why IPO's identity mapping matters.
Explain why Rafailov et al. (NeurIPS 2024) prove DAAs over-optimize despite having no explicit RM.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.