AI Engineering/Phase 19/Lesson 34

Transformer Block from Scratch

One block is the unit of every modern decoder LLM. Layer norm, multi head attention, residual, MLP, residual. The pre-LN variant trains stably without warmup. The post-LN variant is what the original paper shipped. This lesson builds both, side by side, and shows which one survives a 12 layer stack at common learning rates.

Build 1.5 hr Python

Prerequisites: Phase 19 lessons 30 to 33 (tokenizer, embeddings, attention math, batched data loader)

#Learning Objectives

Build a transformer block in PyTorch from the four moving pieces: LayerNorm, multi head causal attention, residual connections, position wise MLP.
Place the LayerNorms in two configurations (pre-LN and post-LN) and explain why one trains stably without warmup.
Implement causal masking inside the multi head attention so token i cannot see tokens j > i.
Track gradient flow through both variants on a 12 layer stack and read the result without hand waving.
Reuse the block as a drop-in unit when the next lesson assembles a 124 million parameter GPT.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.