Flamingo and Gated Cross-Attention for Few-Shot VLMs

DeepMind's Flamingo (2022) did two things before anyone else. It showed a single model could process arbitrarily interleaved sequences of images, videos, and text. And it showed VLMs could learn in-context — give a few-shot prompt with three example (image, caption) pairs and the model captions a new image without any gradient step. The mechanism: gated cross-attention layers, inserted between the frozen LLM's existing layers, with a learned tanh gate that starts at zero so the LLM's text capability is preserved at initialization. This lesson walks Flamingo's Perceiver resampler and gated cross-attention architecture — the ancestor of Gemini's interleaved inputs and Idefics2's visual tokens.

Learn 2 hr Python (stdlib, gated cross-attention + Perceiver resampler demo)

Prerequisites: Phase 12 · 03 (BLIP-2 Q-Former)

#Learning Objectives

Explain how gated cross-attention preserves a frozen LLM's text capability at initialization via tanh(gate) = 0.
Walk through a Perceiver resampler: N image patches → K fixed "latent" queries via cross-attention.
Describe how Flamingo handles interleaved image-text sequences with causal masking that respects image placement.
Reproduce a few-shot multimodal prompt structure (3 image-caption examples then a query image).

#Learning Objectives

Keep going. The rest of this lesson is one step away.