Tokenized Dataset with Sliding Window

A pretraining run is a function from token ids to gradients. This lesson builds the conveyor that feeds the ids in.

Build 1.5 hr Python

Prerequisites: Phase 04 lessons, Phase 07 transformer lessons, Lesson 30 of this phase

#Learning Objectives

Convert a raw corpus into a stream of token ids by calling the tokenizer once.
Slice the id stream into fixed-length windows with a configurable overlap stride.
Build a PyTorch Dataset that returns input and target tensors for next-token prediction.
Wrap the dataset in a DataLoader with a deterministic shuffle seeded per epoch.
Reason about the trade-off between stride, redundancy, and effective dataset size.

Full Access lesson

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.