AI Engineering/Phase 10/Lesson 05

Scaling: Distributed Training, FSDP, DeepSpeed

Your 124M model trained on one GPU. Now try 7 billion parameters. The model doesn't fit in memory. The data takes weeks on a single machine. Distributed training isn't optional at scale. It's the only path forward.

Build 2 hr Python

Prerequisites: Phase 10, Lesson 04 (Pre-Training a Mini GPT)

#Learning Objectives

Explain the three types of parallelism (data, tensor, pipeline) and when each is necessary based on model and cluster size
Implement data-parallel training using PyTorch DDP with gradient synchronization across multiple GPUs
Calculate the memory budget for a given model size (weights + optimizer states + gradients + activations) to determine the minimum hardware
Configure FSDP or DeepSpeed ZeRO stages to shard model states across GPUs and fit models that exceed single-GPU memory

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.