AI Engineering/Phase 19/Lesson 62

Vision-Language Pretraining

The encoder, projection, and decoder are wired. Now train them together. Two objectives drive learning: a contrastive image-text loss (InfoNCE) that pulls matching pairs together in the joint embedding space, and a language modeling loss that asks the decoder to caption each image. Combined, they teach the network both to find the right image for a caption and to write a caption for the image.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 30-37 (Track B foundations)

#Learning Objectives

Implement InfoNCE contrastive loss across a batch of image-caption pairs.
Compose contrastive loss with autoregressive language modeling loss.
Synthesize a 200-pair mock image-caption corpus with no real dataset download.
Run a 50-step demo training loop and observe both losses decreasing.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.