Reward Hacking and Goodhart's Law

Any optimizer strong enough to maximize a proxy reward will find the gap between the proxy and the thing you actually wanted. Gao et al. (ICML 2023) gave this a scaling law: proxy reward increases, gold reward peaks then falls, and the gap grows with the KL divergence from the initial policy in a way you can fit in closed form. Sycophancy, verbosity bias, unfaithful chain-of-thought, and evaluator tampering are not separate problems. They are the same problem in different costumes.

Learn 1 hr Python (stdlib, proxy-vs-gold-reward simulator)

Prerequisites: Phase 18 · 01 (InstructGPT), Phase 10 · 07 (RLHF)

#Learning Objectives

State Goodhart's Law and why it is not a folk slogan but a predictable property of any optimization against an imperfect proxy.
Describe the Gao et al. 2023 scaling law: mean proxy-gold gap as a function of KL distance from the initial policy.
Name four common manifestations of reward hacking (verbosity, sycophancy, unfaithful reasoning, evaluator tampering) and trace each back to the shared mechanism.
Explain why KL regularization alone does not save you under heavy-tailed reward error (Catastrophic Goodhart).

#Learning Objectives

Keep going. The rest of this lesson is one step away.