Quantization: Making Models Fit

A 70B model in FP16 needs 140GB. Two A100s just for weights. Quantize to FP8: one 80GB GPU. INT4: a MacBook.

Build 2 hr Python (with numpy)

Prerequisites: Phase 10, Lessons 01-10 (LLMs from Scratch)

#Learning Objectives

Implement symmetric and asymmetric quantization from FP16 to INT8 and INT4, including per-tensor and per-channel scaling
Calculate the memory savings from quantization and determine which precision fits a given GPU's VRAM
Explain the difference between post-training quantization (PTQ) and quantization-aware training (QAT)
Apply GPTQ or AWQ to quantize a real model and measure the accuracy-memory tradeoff on a benchmark

Full Access lesson

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.