Sign inStart free
AI Engineering/Phase 19/Lesson 46

Gradient Accumulation

Train at an effective batch you cannot afford, one micro-batch at a time. Scale the loss, hold the optimizer step, and let the gradients pile up.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 42 to 45

#Learning Objectives

  • Derive the effective batch identity: effective_batch = micro_batch * accum_steps.
  • Implement loss-per-micro-batch scaling so the accumulated gradient matches a single full-batch backward.
  • Skip optimizer synchronization until the last micro-batch (sync-on-last-step).
  • Read a throughput against effective batch curve and explain the diminishing return.
Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.