Cold Start Mitigation for Serverless LLMs

A 20 GB model image takes 5-10 minutes (7B) to 20+ minutes (70B) to go from cold to serving. In a true serverless world, that is not a warm-up — it is an outage. Mitigations operate at five layers: pre-seeded node images (Bottlerocket on AWS, dual-volume arch), model streaming (NVIDIA Run:ai Model Streamer, native in vLLM), GPU memory snapshots (Modal checkpoints, up to 10x faster restart), warm pools (`min_workers=1`), tiered loading (ServerlessLLM's NVMe→DRAM→HBM pipeline, 10-200x latency reduction), and live migration that moves input tokens (KB) rather than KV cache (GB). Modal publishes 2-4s cold starts as a floor; Baseten 5-10s default, sub-second with pre-warming. This lesson teaches you to measure, budget, and stack the five layers.

Learn 1 hr Python (stdlib, toy cold-start path simulator)

Prerequisites: Phase 17 · 02 (Inference Platform Economics), Phase 17 · 03 (GPU Autoscaling)

#Learning Objectives

Enumerate the five layers of cold-start mitigation and name one tool or pattern at each layer.
Compute total cold-start time as a sum of (node provision) + (weights download) + (weights load into HBM) + (engine init) for a 70B model.
Explain why live migration transfers input tokens (KB) not KV cache (GB) and what the penalty is (recomputation).
Name the warm-pool trade-off (pay for idle GPU or accept cold-start tail) and the SLA threshold at which min_workers > 0 becomes mandatory.

#Learning Objectives

Keep going. The rest of this lesson is one step away.