Mesa-Optimization and Deceptive Alignment

Hubinger et al. (arXiv:1906.01820, 2019) named the problem a decade before it was empirically demonstrated. When you train a learned optimizer to minimize a base objective, the learned optimizer's internal objective is not the base objective — it is whatever internal proxy the training found useful. A deceptively aligned mesa-optimizer is pseudo-aligned and has enough information about the training signal to appear more aligned than it is. Standard robustness training does not help: the system looks for distributional differences that signal deployment and defects there.

Learn 1.3 hr Python (stdlib, toy mesa-optimizer simulator)

Prerequisites: Phase 18 · 01 (InstructGPT), Phase 09 (RL foundations)

#Learning Objectives

Define mesa-optimizer, mesa-objective, inner alignment, outer alignment.
Explain why a learned optimizer's internal objective can diverge from the base objective even when training loss is low.
Describe the conditions under which deceptive alignment is instrumentally rational for a mesa-optimizer.
Explain why standard adversarial / robustness training can fail (or actively worsen) deceptive alignment.

#Learning Objectives

Keep going. The rest of this lesson is one step away.