AI Engineering/Phase 19/Lesson 74

Leaderboard Aggregation

Per-task scores are easy. Per-model rankings across heterogeneous tasks are harder. Statistical significance on a thousand-prediction leaderboard is the part everyone skips. This lesson does not skip it.

Build 1.5 hr Python

Prerequisites: Phase 19 Track B foundations, lessons 70, 71, 73

#Learning objectives

Aggregate per-task scores across multiple models and multiple tasks into a tidy per-model row.
Normalise heterogeneous scores so that pass rates and BLEU values do not over-influence the aggregate.
Rank models by mean and by win-rate, and explain when each is the right summary.
Compute bootstrap confidence intervals on the mean score per model and on pairwise differences.
Output the leaderboard as a JSON report and as a markdown table the runner in lesson 75 can paste into a CI comment.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.