AI Engineering/Phase 04/Lesson 18

Open-Vocabulary Vision — CLIP

Train an image encoder and a text encoder together so that matching (image, caption) pairs land at the same point in a shared space. That is the whole trick.

Build + Use 45 min Python

Prerequisites: Phase 4 Lesson 14 (ViT), Phase 4 Lesson 17 (Self-Supervised)

#Learning Objectives

Explain CLIP's two-tower architecture and contrastive training objective
Use a pretrained CLIP (or SigLIP) for zero-shot classification without any task-specific training
Implement zero-shot classification from scratch: encode class prompts, compute cosine similarity, take argmax
Distinguish CLIP, SigLIP, OpenCLIP, and LLaVA/LLaMA-vision models — what each is for in 2026

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.