AI Engineering/Phase 19/Lesson 73

Perplexity and Calibration

If your model says 90 percent confident on a thousand answers and gets six hundred right, it is not well calibrated. Calibration is half of trustworthy eval. The other half is perplexity, which tells you whether the model thinks the held-out text is plausible at all.

Build 1.5 hr Python

Prerequisites: Phase 19 Track B foundations, lessons 70 and 71

#Learning objectives

Compute token-level perplexity on a held-out corpus from token negative log-probabilities supplied by the model adapter.
Compute the expected calibration error (ECE) of a classifier or multiple-choice eval from binned predicted probabilities.
Compute the Brier score (mean squared error against the indicator of correctness) and explain when it does what ECE does not.
Build the reliability diagram data needed to plot a confidence-versus-accuracy curve.
Wire all three into the eval harness so the runner can attach perplexity, ece, and brier numbers to a model report.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.