Vision Transformers (ViT)

Cut the image into patches, treat each patch as a word, run a standard transformer. Don't look back.

Build 45 min Python

Prerequisites: Phase 7 Lesson 02 (Self-Attention), Phase 4 Lesson 04 (Image Classification)

#Learning Objectives

Implement patch embedding, learned positional embedding, class token, and transformer encoder blocks from scratch to build a minimal ViT
Explain why ViT was thought to need massive pretraining data until DeiT and MAE proved otherwise
Compare ViT, Swin, and ConvNeXt on their architectural priors (none, local window attention, conv backbone)
Fine-tune a pretrained ViT on a small dataset using timm and the standard linear-probe / fine-tune recipe

Full Access lesson

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.