Headline finding
vLLM, built on top of PagedAttention, improves the serving throughput of popular LLMs by 2-4x with the same level of latency compared to state-of-the-art systems such as FasterTransformer and Orca, with more pronounced improvements for longer sequences, larger models, and more complex decoding algorithms, without affecting model accuracy.
The experimental results across Sections 6.2-6.5 consistently demonstrate 1.67x-3.58x (and higher against Orca Max/FasterTransformer) throughput improvements at comparable normalized latencies, and PagedAttention is mathematically equivalent to standard attention.
“Our evaluations on various models and workloads show that vLLM improves the LLM serving throughput by 2-4× compared to the state-of-the-art systems [31, 60], without affecting the model accuracy at all. The improvements are more pronounced with longer sequences, larger models, and more complex decoding algorithms (§4.3).” · Show in paper