AI Engineering/Phase 10/Lesson 08

DPO: Direct Preference Optimization

RLHF works. It also requires training three models (SFT, reward model, policy), managing PPO's instability, and tuning a KL penalty. DPO asks: what if you could skip all of that? DPO directly optimizes the language model on preference pairs. No reward model. No PPO. One training loop. Same results.

Build 1.5 hr Python (with numpy)

Prerequisites: Phase 10, Lesson 07 (RLHF)

#Learning Objectives

Implement DPO training that directly optimizes a language model on preference pairs without a separate reward model
Derive the DPO loss function and explain how it implicitly represents a reward model through the policy's log probabilities
Compare DPO vs RLHF in terms of training stability, compute cost, and number of models required
Tune the beta parameter to control how far the trained policy diverges from the reference model

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.