Training Loop and Evaluation

A loop that does not measure is a loop that lies. This lesson builds the training loop that drives the GPT model: AdamW with weight decay split, a warmup plus cosine learning rate schedule, a `calc_loss_batch` helper, an `evaluate_model` pass on held out data, a `generate_and_print_sample` qualitative probe every K steps, and a JSONL log of losses you can plot after. The same skeleton trains every decoder LLM you will ever build.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 30 to 35

#Learning Objectives

Build a training loop that computes cross entropy loss with the correct input and target alignment for next token prediction.
Configure AdamW with weight decay applied to weight tensors and not to LayerNorm or bias tensors.
Implement a learning rate schedule with linear warmup and cosine decay, and read the resulting LR over time.
Evaluate on a held out split with evaluate_model so the eval loss is comparable across runs.
Generate a qualitative sample every K steps with generate_and_print_sample to catch divergence before the loss curve does.
Persist per step loss to JSONL so you can reload, plot, and ship the training log as a deliverable.

#Learning Objectives

Keep going. The rest of this lesson is one step away.