AI Engineering/Phase 19/Lesson 49

Language Model Evaluation Harness

A model that does well on a task you cannot define is a model that does well by accident. The harness is the task definition, the metric, the runner, and the leaderboard, in one short, swappable shape.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 42 to 45

#Learning Objectives

Define a task as a JSONL file with prompt, targets, metric, and optional extras per example.
Implement five metrics: exact match, rouge-l F1, executable check, multiple choice, and substring contains.
Build a runner that batches examples per task and dispatches to a swappable model adapter.
Emit a leaderboard JSON with per-task scores, latency, and an overall average that is reproducible.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.