Multi-Token Prediction (MTP)

Every autoregressive LLM from GPT-2 to Llama 3 trains on one loss per position: predict the next token. DeepSeek-V3 added a second loss per position: predict the token after that. The extra 14B of parameters (on a 671B model) got distilled back into the main model through gradient flow, and the trained MTP heads were repurposed at inference as speculative-decoding drafters with 80%+ acceptance. 1.8× generation throughput came for free. This lesson builds the sequential MTP module from the DeepSeek technical report, computes the loss and the shared-head parameter layout, and explains why MTP keeps the causal chain while Gloeckle et al.'s original parallel MTP broke it.

Build 1 hr Python (stdlib)

Prerequisites: Phase 10 · 04 (pre-training a mini GPT), Phase 10 · 15 (speculative decoding)

#Learning Objectives

State the MTP training objective and derive the joint loss across prediction depths.
Explain the difference between Gloeckle et al.'s parallel MTP heads (2024) and DeepSeek-V3's sequential MTP modules and why the sequential design preserves the causal chain.
Compute the parameter and memory overhead of adding MTP modules to a pre-training run.
Implement one MTP module from scratch: the shared embedding, the per-depth transformer block, the projection, and the shared output head.

#Learning Objectives

Keep going. The rest of this lesson is one step away.