Sign inStart free
AI Engineering/Phase 07
Advanced

Transformers Deep Dive

Attention, positional encoding and every block of the architecture.

16 lessons ~16.5 hr
  1. Why Transformers — The Problems with RNNsLearn · Free · 45 minRNNs process tokens one at a time. Transformers process all tokens at once. That single architectural bet changed every scaling curve in deep learning after 2017.Open
  2. Self-Attention from ScratchBuild · 1 h 30 minAttention is a lookup table where every word asks "who matters to me?" - and learns the answer.Needs Full Access
  3. Multi-Head AttentionBuild · 1 h 15 minOne attention head learns one relation at a time. Eight heads learn eight. Heads are free. Take more of them.Needs Full Access
  4. Positional Encoding — Sinusoidal, RoPE, ALiBiBuild · 45 minAttention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it — each with a different bet on what "position" means.Needs Full Access
  5. The Full Transformer — Encoder + DecoderBuild · 1 h 15 minAttention is the star. Everything else — residuals, normalization, feed-forward, cross-attention — is the scaffolding that lets you stack it deep.Needs Full Access
  6. BERT — Masked Language ModelingBuild · 45 minGPT predicts the next word. BERT predicts a missing word. One sentence of difference — and half a decade of everything embedding-shaped.Needs Full Access
  7. GPT — Causal Language ModelingBuild · 1 h 15 minBERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI.Needs Full Access
  8. T5, BART — Encoder-Decoder ModelsLearn · 45 minEncoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks: translate, summarize, rewrite, transcribe.Needs Full Access
  9. Vision Transformers (ViT)Build · 45 minAn image is a grid of patches. A sentence is a grid of tokens. The same transformer eats both.Needs Full Access
  10. Audio Transformers — Whisper ArchitectureLearn · 45 minAudio is an image of frequency over time. Whisper is a ViT that eats mel spectrograms and speaks back.Needs Full Access
  11. Mixture of Experts (MoE)Build · 45 minA dense 70B transformer activates every parameter for every token. A 671B MoE activates only 37B per token and beats it on every benchmark. Sparsity is the most important scaling idea of the decade.Needs Full Access
  12. KV Cache, Flash Attention & Inference OptimizationBuild · 1 h 15 minTraining is parallel and FLOP-bound. Inference is serial and memory-bound. Different bottleneck, different tricks.Needs Full Access
  13. Scaling LawsLearn · 45 minThe 2020 Kaplan paper said: bigger model, lower loss. The 2022 Hoffmann paper said: you were under-training. Compute goes into two buckets — parameters and tokens — and the split is not obvious.Needs Full Access
  14. Build a Transformer from Scratch — The CapstoneBuild · 2 hThirteen lessons. One model. No shortcuts.Needs Full Access
  15. Attention Variants — Sliding Window, Sparse, DifferentialBuild · 1 hFull attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost.Needs Full Access
  16. Speculative Decoding — Draft, Verify, RepeatBuild · 1 hAutoregressive decoding is serial. Each token waits for the previous one. Speculative decoding breaks the chain: a cheap model drafts N tokens, the expensive model verifies all N in one forward pass. When the draft is right you paid one big forward for N generations.Needs Full Access