SGLang and RadixAttention for Prefix-Heavy Workloads

SGLang treats the KV cache as a first-class, reusable resource stored in a radix tree. Where vLLM schedules requests FCFS (first-come, first-served), SGLang's cache-aware scheduler prioritizes requests with longer shared prefixes — effectively a depth-first radix traversal so hot branches stay resident in HBM. On Llama 3.1 8B with ShareGPT-like 1K prompts, SGLang hits ~16,200 tok/s to vLLM's ~12,500, a ~29% edge. On prefix-heavy RAG workloads the advantage reaches 6.4x. On voice-cloning-shaped workloads cache hit rate cleared 86%. Deployed on 400,000+ GPUs in 2026 across xAI, LinkedIn, Cursor, Oracle, GCP, Azure, AWS. The gotcha is that the 6.4x number evaporates when prefix ordering is inconsistent — ordering is the engineer's lever.

Learn 1.3 hr Python (stdlib, toy radix-tree cache + cache-aware scheduler)

Prerequisites: Phase 17 · 04 (vLLM Serving Internals), Phase 14 (Agentic RAG)

#Learning Objectives

Diagram RadixAttention: how prefixes are stored in a radix tree and how KV blocks are shared across sequences rooted at the same branch.
Explain cache-aware scheduling and why FCFS is wrong for prefix-heavy traffic.
Compute expected speedup for a workload given prefix-cache hit rate and prompt length distribution.
Name the prompt-ordering discipline that makes the 6.4x number real vs a lost upside.

#Learning Objectives

Keep going. The rest of this lesson is one step away.