Headline finding
SpecStream maintains task quality close to SGLang speculative decoding while supporting more concurrent requests under limited GPU memory, achieving average throughput speedups of 1.41x and 1.32x over the offloading baseline for Qwen3 and InternLM2.5 across different datasets, and improving output throughput per GPU by an average of 55.4% compared with parallel speculative decoding on separate Target and Draft GPUs.
All components of the headline claim—task quality preservation, concurrency under limited GPU memory, throughput speedups over TriForce, and per-GPU throughput gains over separate-GPU parallel speculation—are directly supported by the reported experimental results.
“Experiments show that SpecStream maintains task quality close to SGLang speculative decoding while supporting more concurrent requests under limited GPU memory. Across different datasets, it achieves average throughput speedups of 1.41× and 1.32× over the offloading baseline for Qwen3 and InternLM2.5, respectively. Compared with parallel speculative decoding on separate Target and Draft GPUs, SpecStream improves output throughput per GPU by an average of 55.4%.” · Show in paper