AI Engineering/Phase 19/Lesson 27

Capstone Lesson 27: Eval Harness with Fixture Tasks

A coding agent is only as good as the suite of tasks you measure it against. This lesson builds an evaluation harness that takes a folder of fixture tasks, runs each through a candidate agent, scores pass or fail through a deterministic verifier, and aggregates the results into pass@1, pass@k, mean latency, and mean cost. The harness is the source of truth that lets you tell a regression from a refactor.

Build 1.5 hr Python (stdlib)

Prerequisites: Phase 19 · 25 (verification gates), Phase 19 · 26 (sandbox runner), Phase 14 · 30 (eval-driven agent development), Phase 14 · 19 (SWE-bench and GAIA benchmarks)

#Learning Objectives

Define a fixture task as a triple of goal, setup, and verifier.
Score multiple sample runs per task and compute pass@1 and pass@k.
Aggregate latency and cost into mean and 95th-percentile metrics.
Wire deterministic verifiers (file diff, exit code, regex match) into reusable functions.
Emit a structured JSON report a regression-tracking script can ingest.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.