DeepSeek-V3 Architecture Walkthrough

Phase 10 · Lesson 14 named the six architectural knobs every open model turns. DeepSeek-V3 (December 2024, 671B parameters total, 37B active) turns all six and adds four more: Multi-Head Latent Attention, auxiliary-loss-free load balancing, Multi-Token Prediction, and DualPipe training. This lesson reads DeepSeek-V3's architecture top to bottom and derives every parameter count from the published config. By the end you can explain why the 671B/37B ratio is the right bet and why MLA + MoE together beat either alone at the frontier.

Learn 1.3 hr Python (stdlib, parameter calculator)

Prerequisites: Phase 10 · 14 (open-model walkthroughs), Phase 10 · 17 (NSA), Phase 10 · 18 (MTP), Phase 10 · 19 (DualPipe)

#Learning Objectives

Read the DeepSeek-V3 config top to bottom and explain each field in terms of the six GPT-2 knobs plus four DeepSeek-specific additions.
Derive the total parameter count (671B), active parameter count (37B), and the components that contribute to each.
Compute the KV cache footprint of MLA at 128k context and compare to what a same-active-param dense model with GQA would pay.
State the four DeepSeek-specific innovations (MLA, MTP, auxiliary-loss-free routing, DualPipe) and name which part of the architecture/training stack each one targets.

#Learning Objectives

Keep going. The rest of this lesson is one step away.