AI Engineering/Phase 01/Lesson 09

Information Theory

Information theory measures surprise. Loss functions are built on it.

Learn 1 hr Python

Prerequisites: Phase 1, Lesson 06 (Probability)

#Learning Objectives

Compute entropy, cross-entropy, and KL divergence from scratch and explain their relationship
Derive why minimizing cross-entropy loss is equivalent to maximizing log-likelihood
Calculate mutual information between features and a target to rank feature importance
Explain perplexity as the effective vocabulary size a language model chooses from

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.