Native Sparse Attention (DeepSeek NSA)

At 64k tokens, attention eats 70-80% of decode latency. Every open-model lab has a plan to fix it. DeepSeek's NSA (ACL 2025 best paper) is the one that stuck: three parallel attention branches — compressed coarse-grained tokens, selectively retained fine-grained tokens, and sliding windows for local context — combined through a learned gate. It is hardware-aligned (kernel-friendly), natively trainable (works in pre-training, not bolted on at inference), and on 64k decodes it runs faster than FlashAttention while matching or beating full attention quality. This lesson builds the three branches end-to-end and shows why the sparsity is end-to-end differentiable.

Build 1 hr Python (stdlib)

Prerequisites: Phase 7 · 12 (KV cache, flash-attention), Phase 7 · 15 (attention variants), Phase 10 · 16 (differential attention)

#Learning Objectives

State the three NSA attention branches and what each one captures.
Explain why NSA is "natively trainable" where prior sparse-attention methods were inference-only.
Compute the attention compute savings of NSA versus full attention at 64k context as a function of compression block size and selection top-k.
Implement the three-branch combination in stdlib Python on a short synthetic sequence and verify the gating weights behave.

#Learning Objectives

Keep going. The rest of this lesson is one step away.