Search publications
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Chat stays below it, peaking at OIeff = 181 (Table 12), remaining memory-bandwidth-bound across the full concurrency sweep. The other agentic workloads cross the ridge: computer-use reaches OIeff = 886 (3.0x the ridge), terminal reaches 1,809 (6.1x), and coding agent reaches 2,409 (8.2x) at C = 80, all compute-bound. Peak separation between coding agent and chat is 13x.” · Show in paper
Verified
Table 12 and Figure 4a directly substantiate the exact OI values and ridge multiples on H100 for LLaMA-3.1-8B.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“On average, prompt-based methods lose more than 50% of their baseline accuracy at longer context windows (≥ 32k), and nearly all prompting methods begin to deteriorate sharply after 8k to 16k tokens.” · Show in paper
Verified
Table 1 shows substantial accuracy drops for all four prompt-only baselines between 0k and 32k-128k.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“As shown in Table 3, SmoothQuant can match the FP16 accuracy on all evaluation datasets with all quantization schemes. LLM.int8() can match the floating point accuracy because they use floating-point values to represent outliers, which leads to a large latency overhead (Table 11). The W8A8, ZeroQuant, and Outlier Suppression baselines produce nearly random results, indicating that naively quantizing the activation of LLMs will destroy the performance.” · Show in paper
Verified
Table 3 provides direct empirical evidence that SmoothQuant-O1/O2/O3 closely match FP16 accuracy and perplexity on OPT-175B while the three baselines degrade to near-random performance.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“Figure 7 shows the ITL and TTFT measured on both Llama 3.1 (Dubey et al., 2024) 8B (on 1xH100) and Llama 3.1 70B (on 4xH100) models. Compared to SGLang with Triton backend, FlashInfer backend shows consistent speedup in all settings.” · Show in paper
Verified
Figure 7 clearly shows lower median ITL and TTFT for SGLang (FlashInfer 0.2) than SGLang (Triton) across all four model-workload combinations.
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“On the ShareGPT dataset, vLLM can sustain 1.7×–2.7× higher request rates compared to Orca (Oracle) and 2.7×–8× compared to Orca (Max), while maintaining similar latencies. ... Compared to FasterTransformer, vLLM can sustain up to 22× higher request rates, as FasterTransformer does not utilize a fine-grained scheduling mechanism and inefficiently manages the memory like Orca (Max).” · Show in paper
Verified
The latency-throughput curves in Figure 12 and batch size measurements in Figure 13 directly substantiate the claimed speedups.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model
PaperCodeSource excerpt
“Increasing rank from r=2 to r=16 produced consistent accuracy gains, from 38.6% to 59.6% exact-match (Table 2). Beyond r=16, accuracy plateaued sharply: r=32 achieved only 60.4%, a gain of just 0.8 points despite doubling trainable parameters from 589,824 to 1,179,648. Peak memory remained nearly flat across all rank configurations (1.59–1.61 GB), confirming that for a 60M-parameter model the adapter weights are negligible relative to the frozen base.” · Show in paper
Verified
Table 2 and Table 3 confirm the accuracy plateau beyond r=16 and the negligible memory difference (1.59–1.61 GB) across rank configurations.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“Online latent compaction is nearly lossless, reaching 90.52% accuracy compared with 92.46% for full-context generation, while being highly efficient: it reduces the average KV length by 3.1x, the average per-problem peak by 3.6x, and the maximum peak by 7.9x: the longest full-context trajectories reach the 32,768-token limit, whereas no compacted trajectory exceeds 4,135 tokens.” · Show in paper
Verified
The full 5,000-problem MATH test set evaluation under matched prompts, context limits, and scoring directly supports the accuracy and KV-reduction claims.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“DBTrimKV adds a small overhead compared to TrimKV due to variable-length, head-specific cache handling; however, it still outperforms vanilla inference at scale.” · Show in paper
Verified
Figures 7, 10, 11, and 12 demonstrate that DBTrimKV is substantially faster than Vanilla at scale while incurring a modest overhead relative to TrimKV.
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
PaperCodeSource excerpt
“We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 ± 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in every one of thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 ± 2.0% faster. Its accuracy is level with QLoRA on every model and within a point of fp16 LoRA. The memory premium depends on how the architecture packs its layers: 0.06 to 0.8 GiB on the 1–4B models, 0.5 GiB at 7B, 1.9 GiB at 14B.” · Show in paper
Verified
The accuracy, memory, and speed figures match the reported tables, and the authors appropriately scope the timing claims to the five timed models (1.4B to 7B).
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
PaperCodeSource excerpt
“We present QATFactory, an open-source framework for deployment-aligned quantization-aware distillation (QAD) and reinforcement learning (QARL). QATFactory simulates deployment-time quantization while performing matrix multiplications in BF16, allowing models to adapt to quantization noise without requiring training hardware that natively supports the target format; for example, it supports NVFP4 training on H100 GPUs, which lack FP4 Tensor Cores. The framework supports NVFP4, MXFP4, and llama.cpp’s Q4_K format; dense and mixture-of-experts models; and both full-parameter and LoRA-based training. It exports checkpoints directly to vLLM and llama.cpp without an additional lossy conversion step or added inference overhead.” · Show in paper
Verified
The manuscript thoroughly documents the framework design, supported formats, training workflows, and direct evaluation of exported checkpoints in vLLM and llama.cpp.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“The difference between fake “2bit (K per-channel, V per-token)” quantization and KIVI is that KIVI maintains a full precision key and value cache sliding window for the local relevant tokens. This sliding window is crucial to maintaining accuracy for hard generation tasks such as mathematical reasoning.” · Show in paper
Verified
Table 3 clearly shows that fake 2bit (K-C, V-T) degrades sharply on GSM8K, while KIVI-2 recovers most of the 16bit baseline performance.
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“Specifically, PagedAttention partitions the KV cache of each sequence into KV blocks. Each block contains the key and value vectors for a fixed number of tokens, which we denote as KV block size (B).” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“As shown in Figure 9, the best static split factor varies with different sequence lengths and token budgets, while adaptive compression decomposition provides comparable accuracy to the best static split factor in most cases.” · Show in paper
Verified
Figure 9 shows that adaptive RocketKV closely matches or exceeds the best static split factor across the tested benchmarks and token budgets.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“Based on the figure, we can see that RocketKV consistently outperforms all other methods, especially under lower token budgets. For Llama3.1-8B-Ins, RocketKV achieves almost no accuracy loss with a token budget of 512 and above, and only 1.1% average accuracy drop with a token budget of 256.” · Show in paper
Verified
Table 2, Table 10, Table 16, and Figure 5(a-c) confirm RocketKV's consistent advantage at low token budgets and the 1.1% (52.2 to 51.1) drop at budget 256 on Llama3.1-8B-Ins.
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“The KV block manager also maintains block tables—the mapping between logical and physical KV blocks of each request. Each block table entry records the corresponding physical blocks of a logical block and the number of filled positions. Separating logical and physical KV blocks allows vLLM to dynamically grow the KV cache memory without reserving it for all positions in advance” · Show in paper
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage.” · Show in paper
Verified
The accuracy, TTFT, and peak memory benchmarks across RULER, LongBench, LongHealth5, and GSM8K support the claim that LCLMs improve the Pareto frontier over KV cache compression baselines.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Adding a shared 1024-token synthetic prefix brings deep-turn E2EL MAPE to 3.2% and aggregate E2EL MAPE to 2.4%, within the observed noise scale. Hence synthetic distributional profiles in this paper use this prefix-aware configuration.” · Show in paper
Verified
Table 11 directly reports the ablation results confirming 2.4% aggregate and 3.2% deep-turn E2EL MAPE.
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning· Method
PaperCodeSource excerpt
“AQLoRA-q is the quality setting: adapters on every layer (α=1.0). AQLoRA-s is the speed setting: depth placement with α=.85 and early backward stopping.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2–3× compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3× and declines by only 0.90 and 2.40 pp at 4× and 5×, respectively.” · Show in paper
Verified
The experimental results in Table 1, Tables 4-7, and Table 9 support the 2-3x near-lossless claim across the tested models and tasks, noting that LLaMA-3.1-8B and Qwen2.5-14B used FlashJoLT.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks· Method
PaperCodeDataSource excerpt
“We conduct all our experiments using the open-weight Llama 3.1-Instruct model, which is an instruction-finetuned version of Llama 3.1 (Meta AI 2024). ... For our RAG experiments, we make use of an off-the-shelf and a specialised multilingual models – Jina-reranker (JinaAI) (Multilingual 2024) and paraphrase-multilingual-mpnet-base-v2 (MPNet) (Reimers and Gurevych 2020), respectively.” · Show in paper