Intuition: what the formula does and why

The core idea

Every vLLM scheduler step does a mix of GPU compute, memory reads, and communication. The formula predicts step time by summing physically-motivated time estimates for each resource, each corrected by a learned scalar:

StepTime = β₁ · max(compute, bandwidth)_prefill
         + β₂ · max(compute, bandwidth)_decode
         + β₃ · weight_load_static
         + β₄ · weight_load_moe
         + β₅ · tp_allreduce
         + β₆ · num_requests_per_step
         + β₇

Why this works

Each feature (F_pf_compute, F_dc_kv, etc.) is calculated as "how long would this take at 100% hardware utilization?" — FLOPs divided by peak TFlops, or bytes divided by peak bandwidth. These are always optimistic (real hardware never hits peak), so the β coefficients correct for reality (β > 1 means "hardware runs at 1/β efficiency").

The max(compute, bandwidth) per phase is the roofline model: the GPU is bottlenecked by whichever resource saturates first. Prefill is typically compute-bound (large matrix multiplies); decode is typically memory-bound (reading KV cache for each token). The formula handles both regimes with the same structure.

How it generalizes across architectures

The formula is a sum of components that zero out when they don't apply:

A single set of 7 βs trained on mixed-architecture data captures all three families. The physics features encode what's different about each architecture; the βs encode how efficient the hardware is at each type of work.

The key insight vs. a blackbox model

A blackbox model (e.g., StepTime ~ f(num_prefill, num_decode, num_requests)) learns correlations specific to one model on one GPU. The roofline features inject causal structure: if you double the hidden dimension, the formula knows compute doubles and weight loading doubles — without retraining. The βs only need re-fitting when the hardware changes (and even then, 2-scalar few-shot calibration often suffices).

Test plan review results

Ran 5 parallel verification agents against the design documents, cross-referencing:

• vLLM v0.8+ source code (deepseek_v2.py, llama.py, fused_moe/layer.py)
• DeepSeek-V2 paper (arXiv 2405.04434) and DeepSeek-V3 paper (arXiv 2412.19437)
• DeepSeek-V3 config.json on HuggingFace
• inference-perf repo config schema and examples
• Published roofline literature (Pope et al. 2022, DuetServe, Vidur, JAX Scaling Book)

Verification scope

Key findings

All physics/math is correct:

• Every FLOP formula, bandwidth formula, and weight loading formula verified
• All 7 worked example values match exact arithmetic
• All DeepSeek-V3 config parameters confirmed against HuggingFace
• MLA absorption mechanics verified against DeepSeek-V2 paper and vLLM implementation
• ~57× KV compression ratio confirmed (32,768 / 576 = 56.9×)
• Only approximation: Q projection as d² (~5% overestimate, explicitly documented as S8)

11 issues found and fixed in f768782: