AI Engineering/Phase 19/Lesson 75

End-to-End Eval Runner

Five lessons of plumbing, one lesson to glue them. The runner reads the task spec from lesson 70, calls a model through an adapter, scores with lessons 71 and 72, attaches the calibration report from lesson 73, and emits the leaderboard from lesson 74. Demo self-terminates.

Build 1.5 hr Python

Prerequisites: Phase 19 Track B foundations, lessons 70 through 74

#Learning objectives

Define a ModelAdapter interface that any model (mock, local, API) can satisfy with a small method surface.
Run the eval over a fixture JSONL file with parallel task execution across a worker pool.
Compose the metric layer (exact_match, F1, BLEU-4, ROUGE-L, code_exec) with the calibration layer in one pass.
Emit per-model EvalRun records and feed them straight into the leaderboard aggregator.
Output both a JSON report and a markdown table; self-terminate with exit zero on a clean run, non-zero on validation or runtime failure.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.