Weight Initialization and Training Stability
Initialize wrong and training never starts. Initialize right and 50 layers train as smoothly as 3.
Prerequisites: Lesson 03.04 (Activation Functions), Lesson 03.07 (Regularization)
#Learning Objectives
- Implement zero, random, Xavier/Glorot, and Kaiming/He initialization strategies and measure their effect on activation magnitudes through 50 layers
- Derive why Xavier init uses Var(w) = 2/(fan_in + fan_out) and Kaiming uses Var(w) = 2/fan_in
- Demonstrate the symmetry problem with zero initialization and explain why random scale alone is insufficient
- Match the correct initialization strategy to the activation function: Xavier for sigmoid/tanh, Kaiming for ReLU/GELU
Keep going. The rest of this lesson is one step away.
Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.
The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.