Cadre

From CLIP to BLIP-2 — Q-Former as Modality Bridge

CLIP aligns image and text but cannot generate captions, answer questions, or hold a conversation. BLIP-2 (Salesforce, 2023) solved that with a small trainable bridge: 32 learnable query vectors attend over a frozen ViT's features via cross-attention, then slot directly into a frozen LLM's input stream. 188M parameters of bridge connected an 11B LLM to a ViT-g/14. Every adapter-based VLM through 2026 — MiniGPT-4, InstructBLIP, LLaVA's cousins — is a descendant. This lesson reads the Q-Former's architecture, explains its two-stage training, and builds a toy version that feeds visual tokens into a frozen text decoder.

Build 3 hr Python (stdlib, cross-attention + learnable-query demo)

Prerequisites: Phase 12 · 02 (CLIP), Phase 7 (Transformers)

#Learning Objectives

Explain why a trainable bottleneck between a frozen vision encoder and frozen LLM beats end-to-end finetuning in cost and stability.
Implement a cross-attention block where a fixed set of learnable queries attend to external image features.
Walk through BLIP-2's two-stage pretraining: representation (ITC + ITM + ITG) then generative (LM loss with frozen decoder).
Compare Q-Former to the simpler MLP projector used in LLaVA and argue when each choice wins.

#Learning Objectives

Keep going. The rest of this lesson is one step away.