By Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj, Renee St. Amant, Victor RühleSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-02043.git
Cite
Version link
https://openversions.com/index/arxiv-2607-02043/commit/0dba223665c6dc3d539b3a756748d316e92d0e05
Version identifier (commit hash)
0dba223665c6dc3d539b3a756748d316e92d0e05

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, Kairos reduces P95 TTFT by up to 81% and raises SLO attainment by up to 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.

On a 2P2D A100 cluster running DeepSeek-V2-Lite, utilization is complementary across prefill and decode nodes and accentuated by workload characteristics: prefill SM utilization is 75-84% while decode SM utilization never exceeds 27%.

Prefill compute is not the bottleneck for tail TTFT in disaggregated serving: prefill execution accounts for only 2-23% of P95 TTFT, while the remaining 77-98% is prefill queue wait and inter-node KV-cache transfer.

Small prefill chunks (\chi \le 512) can be safely deflected to decode nodes without violating the 70 ms TBT SLO (increasing step latency by at most 30 ms), whereas larger chunks (\chi \ge 1024) accumulate substantial overhead across chunks and require workload-aware adaptive chunking.

Static deflection policies are not ideal for deflection decisions under bursty workloads: under-deflection misses up to 36% of Oracle's TTFT gain, while over-deflection violates TBT SLOs.

The analytical step-latency predictors achieve MAPE < 10% on held-out traces from Table 1, enabling O(1) per-step latency predictions via simple linear regressors.

Methods

Workload stratification and steady-state profiling on a 2P2D NVIDIA A100 80 GB cluster running DeepSeek-V2-Lite, using NVIDIA Nsight Systems to measure SM, Tensor Core, and HBM bandwidth utilization.

Source excerpts

“We run each workload of Table 1 to steady state on the 2P2D cluster and collect (i) per-node SM, Tensor Core, and HBM bandwidth utilization via NVIDIA Nsight Systems (Table 2) and (ii) per-request P95 TTFT breakdown into prefill queue wait, prefill execution, KV-cache transfer, and decode queue wait (Table 3).” · Show in paper

Controlled microbenchmark measuring decode step latency increase (\Delta T_{\text{step}}(\chi)) across five successive chunked prefill injections while varying decode batch size B, cached context length K, and total batch tokens \chi.

Source excerpts

“We perform a controlled microbenchmark in which we fix a decode batch of size B, with each decode request having K cached tokens, where K corresponds to the median prompt length of the workload (Table 1). We then inject chunked prefill tokens, increasing the total tokens processed per batch to chi, and measure the resulting step latency increase delta T_step(chi) across five successive chunks.” · Show in paper

One-time deployment-time step-latency profiling (~30 minutes) used to train scikit-learn Linear Regression analytical models for prefill, decode, and mixed-batch step latency, combined with a Splitwise-style KV-cache transfer time model.

Source excerpts

“At deployment time, a one-time profiling script collects data for the analytical models by running production traces at various request rates against the target model hardware combination. The step latency and batch contents are recorded at each step and used to train the analytical models. Total profiling time is ~30 minutes per model/hardware combination.” · Show in paper

Kairos deflection decision algorithm combining a Prefill-Node TTFT Estimator, a Decode-Node Feasibility Checker that greedily constructs a TBT-safe chunk schedule, and an \alpha-margin deflection condition.

Source excerpts

“Algorithm 1 composes the estimator and the feasibility checker. For request r, Algorithm 1 (i) estimates the prefill-node TTFT, (ii) builds a TBT-safe chunk schedule on every decode node and computes the resulting decode-node TTFT, (iii) picks the fastest feasible node, and (iv) deflects only when the alpha margin condition is met.” · Show in paper

System implementation on vLLM v0.18.1 (~2000 lines of Python) with a Deflection Dispatcher service refreshed by 100 ms node heartbeats and a chunked prefill scheduler on decode nodes.

Source excerpts

“KAIROS is implemented on top of vLLM v0.18.1 [8] which supports disaggregated prefill/decode through a KV transfer API. KAIROS adds approximately 2000 lines of new code across the profiling for step latency models, the Deflection Dispatcher (which contains the analytical models and the deflection algorithm), the modifications to the scheduler on the decode nodes and plugins to report node state to the Deflection Dispatcher.” · Show in paper