LLaVA and Visual Instruction Tuning

LLaVA (April 2023) is the most copied multimodal architecture on the planet. It replaced BLIP-2's Q-Former with a 2-layer MLP, replaced Flamingo's gated cross-attention with naive token concatenation, and trained on 158k visual-instruction turns generated by GPT-4 from text-only captions. Any practitioner who built a VLM between 2023 and 2026 built some variant of LLaVA. LLaVA-1.5 added AnyRes. LLaVA-NeXT bumped resolution. LLaVA-OneVision unified image, multi-image, and video in one recipe. This lesson reads the recipe, implements the projector, and explains why "simpler won."

Build 3 hr Python (stdlib, projector + instruction-template builder)

Prerequisites: Phase 12 · 02 (CLIP), Phase 11 (LLM Engineering — instruction tuning)

#Learning Objectives

Build a 2-layer MLP projector that maps ViT patch embeddings (dim 1024) to an LLM's embedding dim (dim 4096).
Walk the LLaVA two-stage recipe: (1) projector alignment on 558k caption pairs, (2) visual instruction tuning on 158k GPT-4-generated turns.
Construct a LLaVA-format prompt with the image token placeholder, system prompt, and user/assistant turns.
Explain why the community moved from Q-Former to MLP despite Q-Former's token-budget win.

#Learning Objectives

Keep going. The rest of this lesson is one step away.