Emu3: Next-Token Prediction for Image and Video Generation

BAAI's Emu3 (Wang et al., September 2024) is the 2024 result that should have ended the diffusion-versus-autoregressive debate. A single Llama-style decoder-only transformer, trained only on the next-token-prediction objective, across a unified vocabulary of text + VQ image tokens + 3D VQ video tokens, beats SDXL on image generation and LLaVA-1.6 on perception. No CLIP loss. No diffusion schedule. Classifier-free guidance is used at inference for quality, but the core training objective is next-token prediction with teacher forcing. Published in Nature. This lesson reads the Emu3 thesis — why a better tokenizer plus scale is all you need — and contrasts with diffusion approaches.

Learn 2 hr Python (stdlib, 3D video tokenizer math + autoregressive sampler skeleton)

Prerequisites: Phase 12 · 11 (Chameleon)

#Learning Objectives

Explain why Emu3's single-loss next-token objective works despite the long-held assumption that diffusion is required for image quality.
Describe the 3D video tokenizer: what a spatiotemporal VQ codebook looks like, why patches span time.
Compare Emu3 vs Stable Diffusion XL on (training compute, inference cost, quality ceiling).
Name the three roles the same Emu3 model plays: Emu3-Gen (image gen), Emu3-Chat (perception), Emu3-Stage2 (video gen).

#Learning Objectives

Keep going. The rest of this lesson is one step away.