Transfusion: Autoregressive Text + Diffusion Image in One Transformer

Chameleon and Emu3 bet everything on discrete tokens. They work, but the quantization bottleneck is visible — the image quality plateaus below continuous-space diffusion models. Transfusion (Meta, Zhou et al., August 2024) takes the opposite bet: keep images continuous, drop the VQ-VAE entirely, and train one transformer with two losses. Text tokens get next-token-prediction. Image patches get a flow-matching / diffusion loss. Both objectives optimize the same weights. The architecture underlying Stable Diffusion 3 (MMDiT) is a close cousin. This lesson reads the Transfusion thesis, builds a toy two-loss trainer, and traces the attention mask that lets one transformer do both jobs.

Build 3 hr Python (stdlib, two-loss trainer on MNIST-scale toy)

Prerequisites: Phase 12 · 11 (Chameleon), Phase 8 (Generative AI)

#Learning Objectives

Wire a transformer that runs two losses (NTP on text tokens, diffusion MSE on image patches) on one backbone.
Explain why bidirectional attention across image patches plus causal attention over text tokens is the right mask choice.
Compare Transfusion-style (continuous images, diffusion loss) to Chameleon-style (discrete images, NTP) on compute, quality, and code complexity.
Name MMDiT's contribution: modality-specific weights at each block, joint attention at the residual stream.

#Learning Objectives

Keep going. The rest of this lesson is one step away.