Differential Privacy for LLMs

DP-SGD remains the standard — noise-injected gradient updates provide formal (epsilon, delta) guarantees. Overhead in compute, memory, and utility is substantial; parameter-efficient DP fine-tuning (LoRA + DP-SGD) is the common 2025 configuration (ACM 2025). Two bodies of evidence in tension: canary-based membership inference (Duan et al., 2024) reports limited success against language models; training-data extraction (Carlini et al., 2021; Nasr et al., 2025) recovers substantial verbatim memorization. Resolution (arXiv:2503.06808, March 2025): the gap is in what is measured — inserted canaries vs "most extractable" data. New canary designs enable loss-based MIA without shadow models and yield the first nontrivial DP audit of an LLM trained on real data with realistic DP guarantees. Alternatives: PMixED (arXiv:2403.15638) — private prediction at inference time via mixture of experts on next-token distributions; DP synthetic data generation (Google Research 2024). Emerging attack: Differential Privacy Reversal via LLM Feedback — confidence-score leakage.

Build 1 hr Python (stdlib, DP-SGD noise-injection and ε-δ accountant demonstration)

Prerequisites: Phase 01 · 09 (information theory), Phase 10 · 01 (large-model training)

#Learning Objectives

Define (epsilon, delta)-differential privacy and state the DP-SGD recipe.
Explain the 2024-2025 tension: canary MIA vs training-data extraction give different pictures.
Describe PMixED and why inference-time private prediction is an alternative to DP training.
Describe the Differential Privacy Reversal via LLM Feedback attack.

#Learning Objectives

Keep going. The rest of this lesson is one step away.