Capstone Lesson 41: Full Evaluation Pipeline

Training is the part you can monitor with loss curves. Evaluation is the part you have to design. This lesson builds a unified eval pipeline that takes any trained language model, runs four heterogeneous evals on it, aggregates the results into a per-task report, and ships a local mock LLM-as-judge so the loop runs without a network. The four evals cover the dimensions every shipping model needs: language modelling (perplexity), short-form correctness (exact-match), open-form similarity (token F1), and qualitative scoring (judge).

Build 1.5 hr Python (torch, numpy)

Prerequisites: Phase 19 lessons 30-37 (NLP LLM track: tokenizer, embedding table, attention block, transformer body, pre-training loop, checkpointing, generation, perplexity)

#Learning Objectives

Compute held-out perplexity with masked-token accounting on a tiny transformer.
Run an exact-match eval on short-form factual prompts.
Compute token-level F1 between predicted and reference strings with normalisation.
Build a local mock LLM-as-judge that scores model outputs on a 1-5 scale.
Aggregate the four evals into a single weighted report with per-task breakdown.

#Learning Objectives

Keep going. The rest of this lesson is one step away.