AI Engineering/Phase 19/Lesson 79

Pipeline Parallel and Bubble Analysis

Tensor parallelism splits the matrix multiply across ranks. Pipeline parallelism splits the model across ranks, one stage per rank. Microbatches flow through the pipeline. The empty time at the start and end is the bubble; minimising it is the whole craft.

Build 1.5 hr Python

Prerequisites: Phase 19 Track C lessons 42-49

#Learning Objectives

Split a sequential model into N stages and simulate a forward pipeline across N ranks.
Schedule M microbatches through the pipeline using the GPipe schedule (forward-only fill, then backward) and compute the bubble fraction.
Compare bubble against the interleaved 1F1B schedule used in Megatron-LM and PipeDream.
Defend stage assignment: equal compute per stage matters more than equal parameter count per stage.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.