AI Engineering/Phase 19/Lesson 47

Checkpoint Save and Resume

Train interrupts kill runs; checkpoints let them continue. Save model, optimizer, scheduler, loss history, step counter, and RNG state, atomically, so a kill at any moment leaves a valid file on disk.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 42 to 45

#Learning Objectives

Capture the full training state into a single payload that can be reloaded into a fresh process.
Implement atomic save with write-to-temp then rename so a crash never leaves a half-written file.
Restore the RNG state for Python, NumPy, and PyTorch so the post-resume loss matches the uninterrupted baseline.
Build a sharded checkpoint layout for models that no longer fit in a single file, with hash-verified shards and a JSON index.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.