AI Engineering/Phase 19/Lesson 43

HDF5 Tokenized Corpus

The downloaded corpus has to land in a layout the trainer can stream from at line speed. JSONL on disk does not survive 16 dataloader workers. HDF5 with a resizable, chunked integer dataset does. This lesson builds streaming tokenization into a resizable HDF5 dataset, sharded write across multiple files, memory-mapped read at training time, and a sliding-window dataloader that produces fixed-length sequences with the right packing.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 30-37

#Learning Objectives

Stream documents into a resizable HDF5 integer dataset with deterministic chunking.
Shard the write across multiple HDF5 files so failure is bounded and parallelism is possible.
Read tokens back through HDF5's page-cache-backed chunked layout so the dataloader copies into batch buffers only at batch time.
Implement a sliding-window dataloader that emits fixed-length training sequences with explicit packing rules.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.