TensorRT-LLM on Blackwell with FP8 and NVFP4

TensorRT-LLM is NVIDIA-only but it wins on Blackwell. On GB200 NVL72 with Dynamo orchestration, SemiAnalysis InferenceX measured $0.012 per million tokens on a 120B model in Q1-Q2 2026, against $0.09/M on H100 + vLLM — a 7x economic gap. The stack is three floating-point regimes compounded: FP8 stays critical for KV cache and attention kernels because it has the dynamic range they need; NVFP4 (4-bit microscaling) handles weights and activations; multi-token prediction (MTP) and disaggregated prefill/decode add another 2-3x on top. Day-0 model support loads FP4 weights directly without post-training conversion. The catch for 2026 engineering teams: TRT-LLM is a closed NVIDIA stack, so adopting it trades portability for throughput. Run the math on your mix of models and hardware before committing.

Learn 1.3 hr Python (stdlib, toy FP8/NVFP4 memory and cost calculator)

Prerequisites: Phase 17 · 04 (vLLM Serving Internals), Phase 10 · 13 (Quantization)

#Learning Objectives

Explain why FP8 stays critical for KV cache and attention even when weights are in NVFP4.
Compute the HBM footprint of a frontier model under BF16, FP8, and NVFP4 and reason about where the savings come from.
Name the Blackwell-specific features TRT-LLM exploits (day-0 FP4, MTP, disaggregated serving, all-to-all primitives).
Decide when TRT-LLM's NVIDIA-lock is worth the 7x cost gap vs vLLM on Hopper.

#Learning Objectives

Keep going. The rest of this lesson is one step away.