Optimizers

Gradient descent tells you which direction to move. It says nothing about how far or how fast. SGD is a compass. Adam is GPS with traffic data.

Build 1.3 hr Python

Prerequisites: Lesson 03.05 (Loss Functions)

#Learning Objectives

Implement SGD, SGD with momentum, Adam, and AdamW optimizers from scratch in Python
Explain how Adam's bias correction compensates for zero-initialized moment estimates in early training steps
Demonstrate why AdamW produces better generalization than Adam with L2 regularization on the same task
Select the appropriate optimizer and default hyperparameters for transformers, CNNs, GANs, and fine-tuning

Full Access lesson

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.