Task Spec Format An eval harness is only as good as the contract its tasks honour. Freeze the JSONL shape and the metric vocabulary before you write a single scoring function.
Prerequisites: Phase 19 Track B foundations
# Learning objectives
Define a JSONL task record schema that covers arithmetic, multiple-choice, code execution, classification, and free-text summarisation in one shape.
Pin a closed vocabulary of metric names so downstream lessons (71-73) can dispatch on a single field.
Specify few-shot examples and post-processing rules as part of the task, not the runner, so the same prompt produces the same target across models.
Implement a strict validator that rejects malformed records before they reach the runner.
Ship a 10-task fixture set that exercises every branch of the spec so the validator has something real to chew on.
Keep going. The rest of this lesson is one step away. Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.
The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.