Token and Positional Embeddings

Ids are integers. The model wants vectors. Two lookup tables sit between them, and the choice of the positional one shapes what the model can learn.

Build 1.5 hr Python

Prerequisites: Phase 04 lessons, Phase 07 transformer lessons, Lessons 30 and 31 of this phase

#Learning Objectives

Build a token-embedding lookup table that maps vocabulary ids to dense vectors.
Build a learned positional-embedding lookup table indexed by position.
Build a fixed sinusoidal positional embedding indexed by position with no parameters.
Compose token and positional embeddings into a single input for a transformer block.
Contrast learned and sinusoidal embeddings on length generalization and parameter count.

Full Access lesson

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.