Cadre

AI Control — Safety Despite Subversion

Greenblatt, Shlegeris, Sachan, Roger (Redwood Research, arXiv:2312.06942, ICML 2024). Control reframes the safety question: given an untrusted strong model U that may be adversarially optimizing against you, what protocols let you extract useful work while bounding the probability of an unacceptable outcome? Resources are a trusted weaker model T, a budget of trusted human labor H, and the ability to edit U's outputs. Within this frame, Trusted Editing Pareto-dominates Untrusted Monitoring on code-backdoor tasks. The frame inherits worst-case assumptions from security engineering: safety is the probability that an adversarially chosen U cannot cause a concrete unacceptable outcome, measured under red-team pressure that stands in for U.

Learn 1.3 hr Python (stdlib, safety-vs-usefulness frontier on code-backdoor task)

Prerequisites: Phase 18 · 07 (sleeper agents), Phase 18 · 08 (scheming), Phase 18 · 09 (alignment faking)

#Learning Objectives

Define the AI Control threat model: untrusted strong model U, trusted weaker model T, trusted human budget H.
Describe the Trusted Editing protocol and why it Pareto-dominates Untrusted Monitoring on the ICML 2024 code-backdoor task.
Explain the difference between alignment (U is trustworthy) and control (U is not trustworthy and we are safe anyway).
State the three limits of control: humans do not scale, trusted weak models eventually lag, control does not produce trustworthy strong models.

#Learning Objectives

Keep going. The rest of this lesson is one step away.