AI Engineering/Phase 19/Lesson 61

Cross-Attention Fusion

The projection layer aligns one image vector with one caption vector. A real vision-language decoder needs every text token to attend to every patch token, so the model can ground each word in a region. Cross-attention is how that grounding happens. The text queries; the vision keys and values answer. This lesson builds the cross-attention block, the causal text self-attention, and the mask shapes that keep both legal.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 30-37 (Track B foundations)

#Learning Objectives

Implement multi-head cross-attention where the query stream is text and the key/value stream is vision.
Compose a decoder block: causal self-attention + cross-attention + feed-forward.
Get the mask shapes right: causal mask for self-attention, no mask for cross-attention.
Run a forward pass with batched text tokens and a fixed pool of image tokens.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.