CLIP and Contrastive Vision-Language Pretraining

OpenAI's CLIP (2021) proved a single idea big enough to power the next five years: align an image encoder and a text encoder in the same vector space using only noisy web image-caption pairs and a contrastive loss. Zero supervised labels. 400M pairs. The resulting embedding space does zero-shot classification, image-text retrieval, and plugs into every 2026 VLM as its vision tower. SigLIP 2 (2025) replaced softmax with sigmoid and scaled past CLIP at lower cost. This lesson walks the math from InfoNCE to sigmoid pairwise loss and builds the training step in stdlib Python.

Build 3 hr Python (stdlib, InfoNCE + sigmoid loss implementations)

Prerequisites: Phase 12 · 01 (ViT patches), Phase 7 (Transformers)

#Learning Objectives

Derive InfoNCE loss from mutual information and implement a numerically-stable vectorized version.
Explain why sigmoid pairwise loss (SigLIP) scales to batch 32768+ without the all-gather overhead softmax demands.
Run zero-shot ImageNet classification by constructing text templates (a photo of a {class}) and taking argmax over cosine similarity.
Name the four levers CLIP / SigLIP pretraining gives you: batch size, temperature, prompt template, data quality.

#Learning Objectives

Keep going. The rest of this lesson is one step away.