Code Exec Metric Generated code is right when it passes the tests. The eval harness has to extract code, run it without crashing the host, and tally pass-rates honestly. This lesson builds that surface.
Prerequisites: Phase 19 Track B foundations, lessons 70 and 71
# Learning objectives
Extract a code block from a free-form generation in a way that matches the post-process rule from lesson 70.
Execute candidate code in an isolated subprocess with a wall-clock timeout, output cap, and an import denylist.
Score a task as the fraction of supplied assertion strings that pass against the candidate.
Compute pass-at-k for tasks that sample multiple generations from one model.
Treat sandbox crashes, syntax errors, and timeouts as first-class fail modes with distinct exit codes the runner can log.
Keep going. The rest of this lesson is one step away. Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.
The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.