BPE Tokenizer From Scratch
Bytes in, ids out, ids back to the same bytes. Build the tokenizer that every modern text model still starts from.
Prerequisites: Phase 04 lessons, Phase 07 transformer lessons
#Learning Objectives
- Train a Byte-Pair Encoding vocabulary from a raw text corpus by repeatedly merging the most frequent adjacent symbol pair.
- Implement a deterministic merge table and apply it to fresh text to produce a stream of subword ids.
- Round-trip arbitrary UTF-8 input to ids and back without information loss.
- Reserve and protect special tokens (
<|endoftext|>, <|pad|>) so they survive training and decoding.
- Reason about why a byte-level alphabet is the right floor for a general-purpose tokenizer.
Keep going. The rest of this lesson is one step away.
Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.
The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.