Insight 1 · Hardware Foundation
Know your roofline before you optimize
H100: 989 TFLOP/s FP16, 3.35 TB/s HBM3. Ridge = 295 FLOP/byte. Decode is memory-bound; prefill is compute-bound. Every serving optimization is a response to this constraint. Profile the axis before you pick the technique.
⬡ H100 vs A100: 3× attention throughput · NVLink 4.0: 900 GB/s all-to-all
Insight 2 · Memory
The KV cache is the constraint. Everything else is a response
Decode is 95% memory-bandwidth bound — you pay KV read cost every step. PagedAttention cuts fragmentation to ~0%. MQA/GQA cuts KV size 8×. MLA (DeepSeek-V2) cuts it 8× further via low-rank compression. The whole serving stack traces back to this one bottleneck.
⬡ PagedAttention: 20–60% → ~0% waste · Llama-3 70B GQA: 8× KV reduction
Insight 3 · Scheduling
36.9× from scheduling tokens, not requests
Orca's 2022 insight: swap sequences at iteration level — finished slots refill immediately. No change to hardware, model, or quantization. Speculative decoding adds 2–4× latency reduction on top: draft K tokens fast, verify all K in one pass with tree attention. Same quality.
⬡ Orca OSDI 2022: 36.9× vs static · EAGLE: 3× speedup on LLaMA-2 70B
System Stacks · Why They're Co-dependent
You cannot deploy one without the full dependency chain
vLLM
requires PagedAttention + continuous batching + FlashInfer + preemption
SGLang
requires RadixAttention + cascade attention + KV sharing + jump-forward
TensorRT-LLM
requires FP8 / Transformer Engine + in-flight batching + WGMMA + tensor parallelism