Cadre

Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4

Quantization format is not a universal choice — it is a function of hardware, serving engine, and workload. GGUF Q4_K_M or Q5_K_M owns CPU and edge, delivered through llama.cpp and Ollama. GPTQ wins inside vLLM when you need multi-LoRA on the same base. AWQ with Marlin-AWQ kernels delivers ~741 tok/s on a 7B class model with the best Pass@1 at INT4 — the 2026 default for datacenter production. FP8 stays the middle ground on Hopper, Ada, and Blackwell — near-lossless and widely supported. NVFP4 and MXFP4 (Blackwell microscaling) are aggressive and require per-block validation. Two traps bite teams: calibration dataset must match deployment domain, and KV cache is separate from weight quantization — the AWQ lesson "my model is 4 GB now" forgets the 10-30 GB KV cache at production batch sizes.

Learn 1.3 hr Python (stdlib, toy memory and throughput comparison across formats)

Prerequisites: Phase 10 · 13 (Quantization foundations), Phase 17 · 04 (vLLM Serving Internals)

#Learning Objectives

Name the six production quantization formats and their sweet spots in 2026.
Pick a format given hardware (CPU vs GPU, Hopper vs Blackwell), engine (vLLM, TRT-LLM, llama.cpp), and workload (routine chat, reasoning, multi-LoRA).
Compute the weight memory saved and the KV cache left untouched for a chosen format.
Name the calibration-dataset pitfall that degrades quantized models on domain traffic.

#Learning Objectives

Keep going. The rest of this lesson is one step away.