By Fei Li, Song Liu, Shiqiang Nie, Jinyu Wang, Weiguo WuSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2609-33184.git
Cite
Version link
https://openversions.com/index/arxiv-2609-33184/commit/4cc68427cbede57743c35e1758b6d666e0dbf31f
Version identifier (commit hash)
4cc68427cbede57743c35e1758b6d666e0dbf31f

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

SpecStream maintains task quality close to SGLang speculative decoding while supporting more concurrent requests under limited GPU memory, achieving average throughput speedups of 1.41x and 1.32x over the offloading baseline for Qwen3 and InternLM2.5 across different datasets, and improving output throughput per GPU by an average of 55.4% compared with parallel speculative decoding on separate Target and Draft GPUs.

SpecStream maintains output quality close to that of SGLang SD on GSM8K, LongBench v2, and MRCR for Qwen3-32B/0.6B and InternLM2.5-20B-Chat/1.8B-Chat, with the largest score difference being 0.1988 percentage points.

Across GSM8K, LongBench v2, and MRCR with 8 concurrent requests, SpecStream achieves average throughput speedups of 1.41x and 1.32x over TriForce on Qwen3 and InternLM2.5, respectively, and 1.44x and 1.62x over SD+Offload.

Under limited GPU memory with a GPU-first residency policy on LongBench v2, SpecStream alleviates GPU memory limits by offloading historical KV states to CPU memory (reaching 45.50 and 49.50 GiB per GPU at 32 concurrent requests) and achieves average throughput speedups of 1.40x and 1.23x over TriForce on Qwen3 and InternLM2.5, respectively, though retaining the complete Draft KV Cache on the GPUs limits the number of simultaneous requests.

Compared with SPECTRE+Offload on three GPUs (two for Target, one for Draft), SpecStream on two shared GPUs achieves an average throughput speedup of 1.036x and an average improvement of 55.4% in output throughput per GPU across Qwen3 and InternLM2.5.

In the ablation study on 16K inputs, streaming access for the full query group (K2) yields the largest throughput gain, improving throughput over restoring full history before verification (K1) by an average of 83.2% on Qwen3 and 117.1% on InternLM2.5, while the complete streaming verification scheme (K5) achieves 2.23x to 2.51x and 1.73x to 3.09x the throughput of K1 on the two model pairs.

A longer mean accepted length does not guarantee higher throughput when varying verification width qq; SpecStream's dynamic verification width controller balances round time and useful progress, changing throughput by 8.5% to 16.0% on Qwen3 and -1.2% to 9.6% on InternLM2.5 compared to fixed q=4q=4 (K3).

Methods

Commit-aware rollback-free KV tiering partitions the Target KV Cache into Sealed History [0, H) on CPU, Active Tail [H, C_r) on GPU, and Speculative Frontier [C_r, L_r) on GPU, offloading only irrevocably committed history in batches while keeping candidate rollback local to the GPU.

Source excerpts

“The Target KV Cache is partitioned into the following three regions accordingly, Sealed History: [0,H) (CPU) Active Tail: [H,Cr) (GPU) Speculative Frontier:[Cr ,Lr)(GPU) 0 ≤ H ≤ Cr ≤ Lr.” · Show in paper

KV streaming for multi-query verification divides offloaded history into contiguous chunks, shares each transferred chunk across all q verification queries in a round, and uses online softmax accumulators (m, l, a) with double buffering, cross-layer prefetching, Cohort batching, and I/O-aware dynamic verification width selection.

Source excerpts

“SpecStream uses online softmax to maintain normalization state chunk by chunk [17, 18]. For each query and attention head, it stores three accumulators (m, ℓ,a).” · Show in paper

“SpecStream uses two GPU staging buffers and separate transfer and compute streams to overlap H2D transfers with attention computation.” · Show in paper

Target-priority intra-GPU co-execution uses NVIDIA Multi-Process Service (MPS) and Texture Processing Cluster (TPC) masks to run the Draft and Target models concurrently on the same GPUs, admitting one Draft token at a time only when estimated Target latency increase and Draft step duration fit within the latency budget and remaining transfer-wait window.

Source excerpts

“SpecStream uses NVIDIA Multi-Process Service (MPS) [19] to enable concurrent execution of the Target and Draft models in separate processes. ... The system applies a TPC mask in the separate Draft process so that its compute kernels run only on the designated set of TPCs [16], while preserving the Target model’s original access to compute resources.” · Show in paper

“Starting a Draft step under allocation n0 must satisfy conditions: ∆bT(x,n0) ≤ εT and btD(x,n0) + δ ≤ Sb.” · Show in paper

SpecStream is implemented in SGLang 0.5.17 and evaluated using Qwen3-32B/0.6B and InternLM2.5-20B-Chat/1.8B-Chat on two NVIDIA A800 80GB PCIe Gen4 GPUs with 240GB host memory across GSM8K, LongBench v2, and MRCR.

Source excerpts

“We implement SpecStream on SGLang 0.5.17 [21] and evaluate two Target/Draft model pairs, Qwen3-32B/0.6B [14] and InternLM2.5-20B-Chat/1.8B-Chat [15]. The main experiments use two NVIDIA A800 80GB PCIe Gen4 GPUs and 240GB of host memory.” · Show in paper