AI Engineering/Phase 19/Lesson 60

Projection Layer for Modality Alignment

A vision encoder produces image tokens. A text decoder consumes text tokens. The two live in different vector spaces. A small two-layer MLP projects image tokens into the text embedding space, and a cosine alignment loss against a paired caption pulls the two spaces into agreement. That projection is the smallest piece of a vision-language model and the one that matters most for transfer.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 30-37 (Track B foundations)

#Learning Objectives

Build a two-layer MLP projection that maps image features into the text embedding space.
Construct a mock text embedding table (no pretrained tokenizer, no real corpus).
Compute a cosine alignment loss between projected image tokens and a paired caption embedding.
Train the projection alone with a frozen vision encoder and a frozen text table.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.