Chameleon and Early-Fusion Token-Only Multimodal Models

Every VLM we have seen so far keeps images and text separate. Visual tokens come from a vision encoder, flow into a projector, then meet text inside the LLM. The vision and text vocabularies never overlap. Chameleon (Meta, May 2024) asked: what if they did? Train a VQ-VAE that turns an image into a sequence of discrete tokens from a shared vocabulary. Every multimodal document is now one sequence — text tokens and image tokens interleaved, a single autoregressive loss. Side effect: the model can generate mixed-modality outputs — alternating text and image tokens in a single inference call. This lesson reads the early-fusion thesis and builds a toy version end to end.

Build 3 hr Python (stdlib, VQ-VAE tokenizer + interleaved decoder)

Prerequisites: Phase 12 · 05, Phase 8 (Generative AI)

#Learning Objectives

Explain why a shared vocabulary + single loss changes what the model can do.
Describe how a VQ-VAE tokenizes an image into a discrete sequence compatible with a transformer's next-token objective.
Name Chameleon's training-stability tricks: QK-Norm, dropout placement, LayerNorm ordering.
Compare Chameleon vs BLIP-2's Q-Former approach and describe when each is the right choice.

#Learning Objectives

Keep going. The rest of this lesson is one step away.