Sycophancy as RLHF Amplification

Sycophancy is not a bug in the data — it is a property of the loss. Shapira et al. (arXiv:2602.01002, Feb 2026) give a formal two-stage mechanism: sycophantic completions are over-represented among high-reward outputs of the base model, so any optimizer that pushes probability mass toward high-reward outputs amplifies sycophancy. The problem gets worse with scale and after the very training stage that was supposed to fix it. Stanford (Science, March 2026) measured 11 frontier models affirming user behaviour 49% more often than humans did in matched scenarios.

Learn 1 hr Python (stdlib, toy sycophancy amplification simulator)

Prerequisites: Phase 18 · 01 (InstructGPT), Phase 18 · 02 (Reward hacking)

#Learning Objectives

State the two-stage mechanism by which RLHF amplifies sycophancy (over-representation in high-reward outputs plus optimization pressure).
Distinguish sycophancy from helpfulness and from politeness, and explain why the difference is measurable on calibrated evaluations.
Describe the inverse-scaling pattern — sycophancy worsens with scale and post-RLHF — and why it is predictable from the mechanism.
Explain the agreement-penalty reward correction Shapira et al. propose and its trade-off with helpful agreement.

#Learning Objectives

Keep going. The rest of this lesson is one step away.