Search publications
QLoRA: Efficient Finetuning of Quantized LLMs· Method
PaperCodeSource excerpt
“To summarize, QLORA has one storage data type (usually 4-bit NormalFloat) and a computation data type (16-bit BrainFloat). We dequantize the storage data type to the computation data type to perform the forward and backward pass, but we only compute weight gradients for the LoRA parameters which use 16-bit BrainFloat.” · Show in paper
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
PaperCodeSource excerpt
“Finding 1: QATFactory converges stably across model families, parameter counts from 8B to 230B, and NVFP4, MXFP4, and Q4_K. In vLLM and llama.cpp, the exported QAD checkpoints achieve higher average benchmark accuracy and lower mean KL divergence than the RTN and GPTQ post-training quantization baselines.” · Show in paper
Verified
All reported training curves, benchmark accuracies, and KL divergence metrics in Figures 3-4 and Tables 2-3 directly support this claim.
End-to-End Context Compression at Scale· Method
PaperCodeSource excerpt
“An LCLM consists of an encoder with a pooling method that maps token chunks to soft tokens, an adapter to project to the decoder's embedding dimension, and a decoder that consumes the soft tokens as latent context.” · Show in paper
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs· Method
PaperSource excerpt
“We decompose each decoder layer into constituent operations and compute component-level FLOPs and DRAM bytes from model dimensions, precision, tensor-parallel degree, active token count, and batch size. For component j, the operational intensity is OIj = FLOPsj/DRAMBytesj, and the roofline throughput ceiling is: Proof_j = min (Ppeak, OIj x BWpeak), where Ppeak is peak compute throughput (FLOP/s) and BWpeak is peak memory bandwidth (byte/s) for the target GPU. We convert the ceiling into a component latency lower bound, troof_j = FLOPsj/Proof_j, and aggregate component costs to obtain a layer-level roofline bound.” · Show in paper
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention· Method
PaperSource excerpt
“Table 2 assigns NVFP4 to the QK⊤ and dS paths and retains the softmax-sensitive P V , P⊤dO, and dOV ⊤ products in BF16. Quantized views are reused between forward and backward to avoid precision mismatch and repeated memory traffic.” · Show in paper
SoScale: Second-Order Scale Search for FP4 Quantization
PaperSource excerpt
“Combining Hadamard preprocessing with SoScale is most useful for MXFP4, where it gives the best average score for both Qwen3-8B and Qwen3-14B, and it also gives the best NVFP4 result on Qwen3-14B. However, the best row is not always the Hadamard variant: Qwen3-8B under NVFP4 and Qwen3-14B under HiF4 are best with SoScale alone.” · Show in paper
Verified
Every comparison in the claim is directly supported by the bolded and underlined average scores in Table 2.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“It matches exact-recall offloading caches on LongBench and InfiniteBench and trails on synthetic exact retrieval” · Show in paper
Verified
Table 1 supports the claim: SEKV is slightly ahead on LongBench (+0.54) and nearly level on InfiniteBench (-0.25), while trailing on RULER (-3.32) and NIAH (-4.28).
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PaperSource excerpt
“Across all three models and tasks, PagedWeight always reaches high quality with the lowest memory consumption (Figure 5). These results show that selecting routed expert pages to reduce is more effective than using one static quantization precision policy.” · Show in paper
Verified
Figure 5 clearly demonstrates that PagedWeight achieves lower perplexity and higher reasoning accuracy at equivalent or lower memory consumption across all three MoE models and four benchmarks.
SoScale: Second-Order Scale Search for FP4 Quantization
PaperSource excerpt
“Adding recovery to the MSE objective gives mixed gains over ScaleSearch: it is marginal under MXFP4, decreases NVFP4, and improves HiF4. In contrast, full SoScale is best for all three formats, indicating that recovery is most effective when coupled with Hessian-aware scale-code selection rather than a local MSE objective.” · Show in paper
Verified
The numerical results in Table 4 match the authors' characterization for all three formats.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme.” · Show in paper
Verified
Figure 6 and Table 4 on SCBench directly demonstrate that RocketKV-MT outperforms all non-oracle baselines and closely matches Exact-TopK across all tested token budgets.
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
PaperCodeSource excerpt
“Table 4 reports speed over the thirteen kept sessions. AQLoRA-s averages 11.3% faster than QLoRA with a between-session standard deviation of 2.3, and it is faster in every session, by 7.0 to 15.2%; the smallest margin exceeds every floor.” · Show in paper
Verified
The 11.3 ± 2.3% speedup and 7.0% to 15.2% range are directly supported by Table 4 and sensitivity analysis in Table 20.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model· Method
PaperCodeSource excerpt
“Table 1 lists all evaluated configurations; each changes one variable relative to the LoRA r=8, {q,v}, FP16 reference.” · Show in paper
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“FlashInfer’s kernel significant outperforms FlashAttention kernels in uniform and skewed sequence length distributions because of our load-balanced dynamic scheduler (section 3.3.1). FlashInfer’s decode attention outperforms FlashAttention kernels because our versatile tile size selection (section 3.2.2) and FlashAttention use suboptimal tile size for decoding.” · Show in paper
Verified
Figure 8 and the ablation in Tables 6–7 support the performance advantage and attribute it to the load-balanced scheduler and tile size selection.
QQQ: Quality Quattuor-Bit Quantization for Large Language Models
PaperCodeSource excerpt
“Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 ×, 2.10×, and 1.25× compared to FP16, W8A8, and W4A16, respectively.” · Show in paper
Verified
The reported perplexity, zero-shot accuracy, and vLLM throughput benchmarks on LLaMA models support the claimed parity with SOTA quantization methods and the specific speedup figures.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models· Method
PaperCodeSource excerpt
“In both PyTorch Huggingface and FasterTransformer frameworks, we implement INT8 linear modules and the batched matrix multiplication (BMM) function with CUTLASS INT8 GEMM kernels. We simply replace the original floating point (FP16) linear modules and the bmm function with our INT8 kernels as the INT8 model.” · Show in paper
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
PaperCodeSource excerpt
“We also compare our rule against the only deployed alternative we know of: Unsloth’s released dynamic-4bit selection for Qwen2.5-7B, a hand-curated list covering 34 of the 196 linear layers. Substituting their list for ours in an otherwise identical run gives 89.0 on the 8-task average, matching our 89.0.” · Show in paper
Verified
The direct substitution experiment on Qwen2.5-7B supports the parity claim.
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PaperSource excerpt
“Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94× throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.” · Show in paper
Verified
The reported figures match the experimental tables: (35.25 - 9.86)/35.25 = 72.0% memory savings, 130.1/67.1 = 1.94× throughput, (17.0 - 12.2)/12.2 = 39.3% quality gain, and (401.9 - 385.4)/401.9 = 4.1% max throughput loss.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression· Method
PaperCodeSource excerpt
“For an overall compression ratio of c, we define a split factor r so that c is split into c^r for the first stage and c^(1-r) for the second stage, where 0 <= r <= 1. We use the following formula to adaptively determine r: r = min(0.2 + 0.06 * log2(c), 0.8)” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“The benefit of BridgeKV is more pronounced when there is a large gap between the prefill anchor and decode anchor, especially for output-heavy workloads. As it is shown in Table 7, the benefit can go from 1.75× (1.76×) to 33.86× (27.5×) in terms of response time and energy savings.” · Show in paper
Verified
Table 7 directly supports the reported range of response speedups and energy savings.
SoScale: Second-Order Scale Search for FP4 Quantization
PaperSource excerpt
“Across both Qwen3 models and all three FP4 formats, a SoScale variant obtains the best average score in every model-format group, preserving 96.44%–97.17% of the BF16 average accuracy under MXFP4, 98.99%–99.20% under NVFP4, and 98.38%–99.40% under HiF4. Compared with the strongest non-SoScale baseline in each group, the best SoScale variant improves the average score by 0.21 to 0.77 percentage points; relative to local ScaleSearch, the gains are 0.47 to 1.10 points.” · Show in paper
Verified
The reported relative BF16 percentages and improvements over the strongest non-SoScale baselines and ScaleSearch match the exact values in Table 2.