Loading Pretrained Weights

Training a 124 million parameter model from scratch is a budget decision; loading a published checkpoint is a Tuesday. This lesson loads pretrained GPT-2 style weights from a safetensors file into the exact architecture from lesson 35, walks the parameter name mapping piece by piece, and sanity generates a continuation to prove the load worked. No network, no third party loaders, no opaque magic.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 30 to 36

#Learning Objectives

Read a safetensors file with the safetensors Python library and inspect the tensor names and shapes.
Map each pretrained parameter name onto a parameter inside the lesson 35 GPT model.
Handle the two name conventions that differ between published GPT-2 weights and the model in this track: wte/wpe/h.N.attn.c_attn/c_proj and mlp.c_fc/c_proj versus the locally named tok_embed/pos_embed/blocks.N.attn.qkv/out_proj and mlp.fc1/fc2.
Detect and refuse a shape mismatch with a clear error before any weight assignment happens.
Generate a short continuation with the loaded weights and confirm the tokens come from the loaded distribution, not the randomly initialized one.

#Learning Objectives

Keep going. The rest of this lesson is one step away.