Scalable Oversight and Weak-to-Strong Generalization

Burns et al. (OpenAI Superalignment, "Weak-to-Strong Generalization", 2023) proposed a proxy for the superalignment problem: fine-tune a strong model using labels produced by a weaker model. If the strong model generalizes correctly from imperfect weak supervision, current human-scale alignment methods may extend to superhuman systems. Scalable oversight and W2SG are complementary. Scalable oversight (debate, recursive reward modeling, task decomposition) increases the overseer's effective capability so it can keep up with the model under oversight. W2SG ensures the strong model generalizes correctly from whatever imperfect supervision the overseer provides. Debate Helps W2SG (arXiv:2501.13124, January 2025) combines them.

Learn 1 hr Python (stdlib, W2SG gap simulator)

Prerequisites: Phase 18 · 01 (instruction-following), Phase 18 · 10 (AI Control), Phase 09 (RL foundations)

#Learning Objectives

Define scalable oversight and weak-to-strong generalization and explain how they are complementary.
Describe the Burns et al. 2023 experimental setup: fine-tune GPT-4 using labels from GPT-2.
Explain the performance gap recovered (PGR) metric and what it measures.
State the three major scalable-oversight mechanisms (debate, recursive reward modeling, task decomposition) and one strength of each.

#Learning Objectives

Keep going. The rest of this lesson is one step away.