AI Engineering/Phase 10/Lesson 10

Evaluation: Benchmarks, Evals, LM Harness

Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Every frontier lab games benchmarks. MMLU scores go up while models still can't reliably count the number of R's in "strawberry." The only eval that matters is YOUR eval -- on YOUR task, with YOUR data.

Build 1.5 hr Python

Prerequisites: Phase 10, Lessons 01-05 (LLMs from Scratch)

#Learning Objectives

Build a custom evaluation harness that runs multiple-choice and open-ended benchmarks against a language model
Explain why standard benchmarks (MMLU, HumanEval) saturate and fail to differentiate frontier models
Implement task-specific evals with proper metrics: exact match, F1, BLEU, and LLM-as-judge scoring
Design a custom evaluation suite targeting your specific use case rather than relying solely on public leaderboards

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.