Activation Functions

Without nonlinearity, your 100-layer network is a fancy matrix multiply. Activations are the gates that let neural networks think in curves.

Build 1.3 hr Python

Prerequisites: Lesson 03.03 (Backpropagation)

#Learning Objectives

Implement sigmoid, tanh, ReLU, Leaky ReLU, GELU, Swish, and softmax with their derivatives from scratch
Diagnose the vanishing gradient problem by measuring activation magnitudes through 10+ layers with different activations
Detect dead neurons in a ReLU network and explain why GELU avoids this failure mode
Select the correct activation function for a given architecture (transformer, CNN, RNN, output layer)

Full Access lesson

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.