Many-Shot Jailbreaking

Anil, Durmus, Panickssery, Sharma, et al. (Anthropic, NeurIPS 2024). Many-shot jailbreaking (MSJ) exploits long context windows: stuff hundreds of faux user-assistant turns where the assistant complies with harmful requests, then append the target query. Attack success follows a power law in the number of shots; fails at 5 shots, reliable at 256 shots on violent and deceitful content. The phenomenon follows the same power law as benign in-context learning — the attack and ICL share an underlying mechanism, which is why defenses that preserve ICL are hard to design. Classifier-based prompt modification reduces attack success from 61% to 2% on tested settings.

Learn 45 min Python (stdlib, in-context learning vs MSJ simulator)

Prerequisites: Phase 18 · 12 (PAIR), Phase 10 · 04 (in-context learning)

#Learning Objectives

Describe the many-shot jailbreaking attack and the context-window property it exploits.
State the empirical power law: attack success rate as a function of shot count.
Explain why MSJ shares a mechanism with benign in-context learning, and what that implies for defenses.
Describe Anthropic's classifier-based prompt modification defense and its reported 61% -> 2% reduction.

#Learning Objectives

Keep going. The rest of this lesson is one step away.