Search publications
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“As shown in Figure 9 (top), compared to FasterTransformer’s FP16 implementation of OPT, SmoothQuant-O3 can further reduce the execution latency of OPT-13B and OPT-30B by up to 1.56× when using a single GPU. This is challenging since FasterTransformer is already more than 3× faster compared to the PyTorch implementation for OPT-30B. Remarkably, for bigger models that have to be distributed across multiple GPUs, SmoothQuant achieves similar or even better latency using only half the number of GPUs (1 GPU instead of 2 for OPT-66B, 4 GPUs instead of 8 for OPT-175B).” · Show in paper
Verified
Figures 8 and 9 and Table 8 directly report the measured speedups, memory reductions, and GPU count reductions.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“LCLMs achieve the fastest compression time at every context length and substantially lower peak memory at long contexts; all baselines run out of memory at 512K tokens, except Attention Matching, which fails at 512K due to numerical instability in its linear solver and runs out of memory at 1M.” · Show in paper
Verified
Figure 4 and Section 6.2 clearly document the scaling measurements and memory limits on the H200 GPU.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“As shown in Figure 5, with similar maximum memory usage, KIVI enables up to 4× larger batch size and gives 2.35× ∼ 3.47× larger throughput.” · Show in paper
Verified
Figure 5 and Section 4.2.4 support the 4x batch size and 2.35x ~ 3.47x throughput improvement for Llama-2-7B on an A100 80GB GPU.
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
PaperCodeSource excerpt
“Finding 3: LoRA significantly reduces training memory requirements: for Qwen3.5 9B at an 8K sequence length, rank-16 LoRA reduces GPU memory usage by 2.9×. However, LoRA underperforms full-parameter QAD, and increasing the adapter rank does not close the gap.” · Show in paper
Verified
Memory measurements in Table 4 and rank-scaling evaluations in Tables 5 and 6 fully support the claim.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“while transferring 3× less data per decoding step than ShadowKV and using the least total memory among the methods we profile. At 128K context, it halves peak GPU memory and cuts total memory by one third relative to full KV caching.” · Show in paper
Verified
Tables 2 and 20 directly confirm the 3.1x transfer reduction vs. ShadowKV, lowest total memory (46.01 GB), ~50% peak GPU memory reduction, and ~34% total memory reduction vs. Full KV.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model
PaperCodeSource excerpt
“QLoRA INT8 and NF4 configurations achieved 52.8% and 53.2% exact-match accuracy respectively, compared to 53.4% for the FP16 LoRA r=8 baseline with the same {q,v} module set. The accuracy cost of quantization is therefore minimal (0.2–0.6 points), while the memory savings are substantial: both QLoRA configurations require only 0.60 GB peak training memory, a 62% reduction relative to FP16 LoRA (1.59 GB) and a 74% reduction relative to full fine-tuning (2.31 GB). Inference latency is higher for INT8 (921 ms/query) than NF4 (614 ms/query) and FP16 LoRA (around 380 ms/query), reflecting the overhead of de-quantization on a T4 GPU. NF4 represents the best quantization trade-off: near-FP16 accuracy at 0.60 GB memory with 34 tok/s throughput.” · Show in paper
Verified
Table 2 and Section 5.5 support the accuracy, memory reduction, and latency comparisons.
Resource-Efficient Speculative Decoding for Long-Context LLM Serving
PaperSource excerpt
“SpecStream achieves average throughput speedups of 1.40× and 1.23× over TriForce on Qwen3 and InternLM2.5, respectively.” · Show in paper
Verified
Figure 6 and Section 5.2.2 support both the throughput speedups over TriForce and the nuanced limitation regarding Draft KV Cache residency.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“In early experiments, we consider directly training the full model end-to-end from the beginning, but find that this approach underperforms the multi-stage recipe... We also consider parameter-efficient variants that keep the decoder fully frozen (Ge et al., 2023; Cheng et al., 2024; Yen et al., 2024) or use LoRA (Liao et al., 2025; Feldman and Artzi, 2025), but we find them to substantially underperform compared to full-parameter training in early experiments; we therefore focus on end-to-end training. With compression disabled, the trained decoders show minimal drift from the original model (Appendix D.3).” · Show in paper
Verified
Table 6 quantitatively verifies minimal decoder drift when compression is disabled; the early-experiment comparisons against direct end-to-end, frozen-decoder, and LoRA training are reported qualitatively in Section 4.2.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“On Mistral-7B-v0.3 at context length 8,192, peak decode memory of the KV representation is 1.43×, 2.40×, 3.61×, and 4.73× below the dense bf16 cache at the 2×, 3×, 5×, and 8× compression targets.” · Show in paper
Verified
The memory savings and latency overhead are both transparently reported in Section 4.5.
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning· Method
PaperCodeSource excerpt
“All experiments also use a set of shared levers, applied to every method: SDPA attention, LoRA dropout 0, length-grouped dynamic batching, and a fused AdamW.” · Show in paper
Resource-Efficient Speculative Decoding for Long-Context LLM Serving
PaperSource excerpt
“Experiments show that SpecStream maintains task quality close to SGLang speculative decoding while supporting more concurrent requests under limited GPU memory. Across different datasets, it achieves average throughput speedups of 1.41× and 1.32× over the offloading baseline for Qwen3 and InternLM2.5, respectively. Compared with parallel speculative decoding on separate Target and Draft GPUs, SpecStream improves output throughput per GPU by an average of 55.4%.” · Show in paper
Verified
All components of the headline claim—task quality preservation, concurrency under limited GPU memory, throughput speedups over TriForce, and per-GPU throughput gains over separate-GPU parallel speculation—are directly supported by the reported experimental results.
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“The improvement of vLLM over Orca (Oracle) on OPT-13B and the Alpaca dataset goes from 1.3× in basic sampling to 2.3× in beam search with a width of 6. Fig. 15 plots the amount of memory saving, computed by the number of blocks we saved by sharing divided by the number of total blocks without sharing. We show 6.1% - 9.8% memory saving on parallel sampling and 37.6% - 55.2% on beam search. In the same experiments with the ShareGPT dataset, we saw 16.2% - 30.5% memory saving on parallel sampling and 44.3% - 66.3% on beam search.” · Show in paper
Verified
Figures 14 and 15 directly support the Alpaca throughput and memory saving numbers. The ShareGPT memory saving numbers (16.2%-30.5% and 44.3%-66.3%) are reported in text without a corresponding figure, but are consistent with ShareGPT's longer prompts.
QLoRA: Efficient Finetuning of Quantized LLMs
PaperCodeSource excerpt
“Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU.” · Show in paper
Verified
Tables 6 and 7 substantiate the 99.3% relative score and top open-source ranking on the Vicuna benchmark.
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“The attention rows compare BF16 SDPA with the NVFP4 Q/K body; QKV and output projections are counted in the complete-step table. Ours is 1.27× faster and uses 51.1%/52.9% less incremental allocation.” · Show in paper
Verified
All quality, error-analysis, and native-hardware speedup/memory claims for Q-Attn match the reported tables and figures.
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“Across six model pairs and five datasets, BridgeKV outperforms fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings. It closes up to 80% of the quality gap to using the larger model for both phases. On one A40 GPU, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B achieves 3.0× faster requests than the larger model alone, with latency close to the smaller model alone. BridgeKV can improve the hardware and system solutions that separate prefill and decode phases across GPUs. A separate serving simulation models request arrivals, queues, and cache transfer. For Llama-3.1-Instruct-70B→Llama-3.2-Instruct-1B, it predicts up to 33.86× faster requests and 27.5× lower energy use than using the larger model for both prefill and decode phases.” · Show in paper
Verified
All components of the overarching claim are directly supported by Figure 3, Figure 1, Section 4.3, and Table 7.
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention· Method
PaperSource excerpt
“LoRA-SVD parameterizes a training-time weight as W = Wres + βL2L1. It preserves a compact principal subspace and its activation path in BF16 while computing the full-shape residual in NVFP4.” · Show in paper
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
PaperCodeSource excerpt
“To address these challenges, we propose ARCQuant, a framework that boosts NVFP4 performance via Augmented Residual Channels. Distinct from methods that compromise block isolation or hardware uniformity, ARCQuant maintains a strictly unified NVFP4 format by augmenting the activation matrix with quantized residual channels. ... Extensive experiments on LLaMA and Qwen models demonstrate that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks. Furthermore, deployment on RTX 5090 and RTX PRO 6000 GPUs confirms practical benefits, achieving up to 3× speedup over FP16.” · Show in paper
Verified
The experimental results in Table 1, Figure 6, and Table 10 directly support the accuracy and speedup claims on LLaMA and Qwen models across RTX 5090 and RTX PRO 6000 GPUs.
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“As shown in Fig. 18a, this leads to 20–26% higher attention kernel latency, compared to the highly-optimized FasterTransformer implementation. We believe the overhead is small as it only affects the attention operator but not the other operators in the model, such as Linear. Despite the overhead, PagedAttention makes vLLM significantly outperform FasterTransformer in end-to-end performance (§6).” · Show in paper
Verified
Figure 18a and Section 6 end-to-end results support this trade-off.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“The per-component decomposition reveals why: projection GEMMs become compute-bound in prefill (OI up to 2,956 at C=80), while attention, elementwise, and all decode components stay bandwidth-bound (Table 14). The resulting prefill-to-decode OI gap spans 8-28x across workloads (Table 13).” · Show in paper
Verified
Tables 13 and 14a and Figure 2a provide the exact component-level OI values, latency shares, and prefill-to-decode gaps.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.” · Show in paper
Verified
The reported 400x compression ratio, 3.7x A100 speedup, 32.6% peak memory reduction, and near-Full-KV accuracy are directly supported by the experimental results in Section 4.2, Section 4.3, Figure 5, Figure 7, and Table 2.