AI Engineering/Phase 19/Lesson 52

Experiment Runner

The loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json metrics blob the evaluator can trust.

Build 1.5 hr Python

Prerequisites: Phase 19 Track A lessons 20-29

#Learning Objectives

Encode an experiment as a typed spec the runner can serialise to a subprocess.
Launch a subprocess with a hard wall clock timeout and a soft memory cap, and surface both as terminal conditions.
Capture stdout, stderr, and the structured metrics blob into a single result record.
Build an ablation table that sweeps one configuration knob at a time over a fixed base spec.
Keep every result deterministic given a seed so the evaluator sees the same numbers across runs.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.