By Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, Luis CezeSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2501-01005.git
Cite
Version link
https://openversions.com/index/arxiv-2501-01005/commit/53a782e35ea66a6d172dc3406cfc36f4a696e807
Version identifier (commit hash)
53a782e35ea66a6d172dc3406cfc36f4a696e807

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2501.01005v2 OpenVersions: b9666a23-e7b7-48ac-b1cb-4dd8503e4f79

Posted

Claims

Findings

Headline finding

FlashInfer is a customizable and efficient attention engine for LLM serving that achieves 29–69% inter-token-latency reduction compared to compiler backends for LLM serving benchmarks, 28–30% latency reduction for long-context inference, and 13–17% speedup for LLM serving with parallel generation.

Compared to SGLang with a Triton v3.0 backend, integrating FlashInfer v0.2 into SGLang v0.3.4 yields consistent speedups in both Inter-Token-Latency (ITL) and Time-To-First-Token (TTFT) on Llama 3.1 8B (1xH100) and Llama 3.1 70B (4xH100) across ShareGPT and Variable workloads.

FlashInfer's generated kernels significantly outperform FlashAttention2/3 kernels in uniform and skewed sequence length distributions due to its load-balanced dynamic scheduler, and its decode attention outperforms FlashAttention kernels due to versatile tile size selection.

Enabling FlashInfer's load-balancing scheduler reduces ITL and TTFT compared to disabling load-balancing on variable sequence length workloads.

FlashInfer's JIT-compiled fused RoPE+attention kernel achieves 1.6–3.7x higher bandwidth utilization than unfused FlashAttention RoPE and attention kernels, and yields a 28–30% end-to-end ITL reduction in Streaming-LLM on Vicuna-13B (MT-Bench).

For moderate levels of parallel generation (4 <= n <= 32), FlashInfer's composable formats in MLC-Engine yield consistent speedups for both ITL and TTFT on Llama 3.1 8B and 70B using ShareGPT, with peak speedups at n = 4 where ITL decreases by 13.73% (8B) and 17.42% (70B) and TTFT decreases by 16.41% (8B) and 22.86% (70B).

The performance overhead of sparse gathering in FlashInfer (using page size 1) compared to dense contiguous KV-Cache is within 1% for decode kernels and approximately 10% for causal prefill kernels.

FlashInfer consistently outperforms FlexAttention across Causal Attention, Logits SoftCap, ALiBi Bias, and Sliding Window attention variants on NVIDIA H100 80GB SXM, and achieves up to a 20x speedup over PyTorch SDPA and FlexAttention on fine-grained block-sparse KV-cache (Quest) for long sequence lengths.

Methods

Unifying diverse KV-cache storage structures (such as PageAttention, RadixAttention, Tree Attention, and importance masks) under a Block-Sparse Row (BSR) matrix format with arbitrary (B_r, B_c) block sizes, while storing queries, keys, values, and outputs as unpadded ragged tensors.

Source excerpts

“Query and output matrices are efficiently stored as ragged tensors (also known as jagged arrays) (Tensorflow Developers, 2018) without padding, which facilitates the compact packing of queries and outputs from diverse requests into a single tensor.” · Show in paper

“The KV-Cache employs a block-sparse row (BSR) format, where block sizes are defined by application requirements: B_r corresponds to the query tile size, details of which will be discussed in later sections, and B_c is specified by KV-Cache management algorithms. FlashInfer kernel implementations supports arbitrary (B_r, B_c) values.” · Show in paper

Decomposing the KV-cache sparse matrix into multiple BSR submatrices with different block sizes (composable formats) without data movement, using larger B_r for shared prefixes to enable shared-memory and register reuse.

Source excerpts

“Our composable format design allows for the decomposition of the KV cache sparse matrix based on prior knowledge. For instance, if certain requests share a prefix, the corresponding rows and columns in the KV cache form a dense submatrix. We can then use a block sparse matrix with a larger B_r to store these submatrices efficiently.” · Show in paper

Implementing CUDA/CUTLASS templates based on FlashAttention2 (up to Ada) and FlashAttention3 (Hopper) that gather scattered sparse KV-cache tiles into contiguous shared memory via LDGSTS.128B (or TMA for contiguous KV-cache on Hopper) and select FA2 tile sizes using hardware occupancy and workload intensity heuristics.

Source excerpts

“We developed CUDA/CUTLASS (Thakkar et al., 2023) templates for FlashAttention, designed specifically for both dense and block-sparse matrices and compatible with NVIDIA GPU architectures from Turing to Hopper (sm75 to sm90a).” · Show in paper

“We use asynchronous copy instructions LDGSTS with a 128B width to maximize memory bandwidth. Although the Tensor Memory Accelerator (TMA) in Hopper architecture can further accelerate data movement, it doesn’t support non-affine memory access patterns. Consequently, we only use TMA for contiguous KV-Cache on Hopper GPUs and fall back to Ampere-style asynchronous copies for other settings where TMA isn’t suitable.” · Show in paper

Providing a customizable CUDA template and PyTorch-based JIT compiler that accepts user-defined attention variant functors (QueryTransform, KeyTransform, ValueTransform, OutputTransform, LogitsTransform, LogitsMask, and optional softmax) to generate and register custom operators.

Source excerpts

“Inspired by FlexAttention (He et al., 2024), we designed a customizable CUDA template and a JIT compiler that takes the attention variant specification as input and generates the optimized kernel code.” · Show in paper

Using a CPU-side Inspector-Executor (plan/run) load-balanced dynamic scheduler (Algorithm 1) that splits long KV sequences into chunks, deterministically assigns work tiles to CTAs via a min-priority queue, and uses persistent kernels with fixed-offset workspace buffers to maintain CUDAGraph compatibility.

Source excerpts

“FlashInfer guarantees both attention and contraction stage are compatible with CUDAGraphs (Gray, 2019; Nguyen et al., 2021). Both attention and contraction stage use persistent kernel and the grid size is fixed once compiled, which means the kernel is launched with the same grid size for each generation step. We set the fixed offset for each section of the workspace buffer to store partial outputs and plan information to make sure the pointers passed to the kernel are the same for each generation step, meeting the requirement of CUDAGraphs” · Show in paper

Applying head-group fusion for Grouped-Query Attention (GQA) by mapping different KV heads to individual threadblocks while fusing query heads with the query length dimension for short query lengths.

Source excerpts

“To address this limitation, FlashInfer offers a head-group fusion strategy: different KV heads are mapped to individual threadblocks, while query heads are fused with the query length dimension.” · Show in paper