By Fengyao Bai, Hongbin Zhang, Zhitao Chen, Jiangsu Du, Zhiguang Chen, Yutong LuSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2605-23389.git
Cite
Version link
https://openversions.com/index/arxiv-2605-23389/commit/c0dad9cbd5451ad7e9f40f02ac42bf391d53b506
Version identifier (commit hash)
c0dad9cbd5451ad7e9f40f02ac42bf391d53b506

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

AlignedServe improves decoding throughput by a maximum of 1.98x and reduces P99 TPOT latency by up to 7.4x compared to state-of-the-art systems (vLLM, DistServe, and DeepSpeed-FastGen) across synthetic and application workloads.

Tokens generated in the same decode iteration have different costs depending on the lengths of their prefixes (relied KVCache), so mixing a small fraction of tokens with long prefixes into a batch introduces iteration-level bubbles that significantly degrade overall performance.

Grouping requests with similar prefix lengths into a batch via prefix-aware batching substantially reduces forward computing latency and improves decoding throughput compared to prefix-unaware FCFS batching.

Using prefill GPUs to prefetch KVCache from CPU memory and forward it to decode GPUs via NVLink (GPU-Prefetch-For-GPU) significantly reduces iteration scheduling latency compared to direct PCIe transmission.

Disabling GPU prefetching in AlignedServe reduces decoding throughput by 14.73% on AzurePublicDataset, and disabling both GPU prefetching and prefix-aware batching reduces throughput further by 28.51%.

The overheads of batch switching, CPU KV pool memory consumption, and TTFT degradation in AlignedServe remain limited under realistic workloads.

Methods

Prefix-aware batching using a quad-tree and Density First Search (Algorithm 1) to group in-flight requests with similar prefix lengths into batches constrained by maximum memory blocks Bmax⁡B_{\max} and minimum batch size Kmin⁡K_{\min}.

Source excerpts

“We maintain a tuple (request counter, block counter) at each internal node to characterize the corresponding sub-tree, where the request counter denotes the total number of in-flight requests kept in the sub-tree, and the block counter denotes the total number of memory blocks consumed by the KVCache of these in-flight requests.” · Show in paper

“Algorithm 1 Density First Search Algorithm Input: Maximum memory blocks B_max, Minimum batch size K_min Output: Request in a batch B” · Show in paper

Disaggregated GPU-Prefetch-For-GPU architecture with a CPU KV Pool, prefill instances acting as intermediate KVCache buffers (Candidate Batch Buffer and Candidate Requests Buffer), and decoding instances connected via NVLink.

Source excerpts

“The proposed serving system consists of three components, i.e., the KV pool, prefill instances and decoding instances, where both the prefill and decoding instances may be comprised of only one GPU, or multiple GPUs within the same server, or even many GPUs from different servers organized in a pipeline.” · Show in paper

“Instead, we propose to prefetch a prepared batch from host memory to the HBM of prefill instances, and further deliver the batch to decoding instances via NVLink between GPUs upon scheduling.” · Show in paper

Batch-level scheduling policy (Algorithm 2) that schedules requests from the Candidate Requests Buffer or Candidate Batch Buffer and evicts longest or old-batch victims to the Candidate Requests Buffer when HBM is insufficient.

Source excerpts

“Algorithm 2 Batch-Level Scheduling Policy Input: Running batch B_run, candidate requests buffer R_cand, candidate batch buffer B_cand” · Show in paper

Starvation handling using internal-node timestamps in the quad-tree and dynamic scheduling of matching-prefix requests from the KV Pool into the Candidate Requests Buffer.

Source excerpts

“To handle this situation, we set a time stamp for each internal node in the quad-tree, where the time stamp indicates how long the corresponding subtree has not generated a batch. If the time stamp associated with a subtree exceeds a given threshold, we give a higher priority to the subtree, thus avoiding potential starvation.” · Show in paper