Capstone Lesson 40: Direct Preference Optimization from Scratch

Reward models and PPO are the classical RLHF stack. DPO collapses that stack into a single supervised loss that fits a policy directly against preference pairs. This lesson derives the DPO loss from the reward-difference identity, ships a working reference model plus policy model, computes per-token log-probabilities, and trains a tiny transformer on a preference fixture of chosen and rejected completions. Tests pin the loss math and the gradient direction so you know the implementation matches the paper.

Build 1.5 hr Python (torch, numpy)

Prerequisites: Phase 19 lessons 30-37 (NLP LLM track: tokenizer, embedding table, attention block, transformer body, pre-training loop, checkpointing, generation, perplexity)

#Learning Objectives

Derive the DPO loss as a sigmoid over a scaled log-ratio difference and connect it to the implicit reward.
Build a reference model + policy model pair with a frozen reference and a trainable policy.
Compute sequence-level log-probabilities under both models, masking prompt tokens.
Train the policy on (prompt, chosen, rejected) triples and watch the chosen log-prob rise relative to rejected.
Pin behaviour with tests on the loss math, the gradient sign, and the reference invariance.

#Learning Objectives

Keep going. The rest of this lesson is one step away.