Search publications
QLoRA: Efficient Finetuning of Quantized LLMs
PaperCodeSource excerpt
“We present QLORA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.” · Show in paper
Unsupported
While QLoRA's memory reduction to <48GB for 65B models is supported, preserving full 16-bit finetuning performance is only verified up to 3B models; at 7B-65B scales, QLoRA is only compared against 16-bit LoRA, not full 16-bit finetuning, as acknowledged in Section 8.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving· Method
PaperCodeSource excerpt
“We conduct experiments on two widely used instruction-tuned models: Llama-3.1-8B-Instruct [16] and Mistral-7B-Instruct-v0.3 [21].” · Show in paper
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“Hence, we can actually fit the entire quantized model into a single 80GB A100 GPU, which can be executed by dynamically dequantizing layers as they are required during inference ... For example, using our kernels, the 3-bit OPT-175B model obtained via GPTQ running on a single A100 is about 3.25× faster than the FP16 version (running on 5 GPUs) in terms of average time per token. ... executing the 3-bit OPT-175B model on 2x A6000 GPUs reduces latency from 589 milliseconds for FP16 inference (on 8 GPUs) to 130 milliseconds, a 4.5× latency reduction.” · Show in paper
Verified
Table 6 and Appendix A.2.2 provide the exact latency measurements and experimental setup supporting the single-GPU fit and 3.24x/4.53x speedups.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. ... We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000).” · Show in paper
Verified
The experimental results in Tables 2, 5, and 6 directly support the 4-hour quantization time, 3-4 bit accuracy preservation, single-GPU execution on A100, and 3.24x/4.53x speedups.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“As can be seen, GPTQ quantizes 1-3 billion parameter models in a matter of minutes and 175B ones in a few hours.” · Show in paper
Verified
Table 2 provides exact timing measurements confirming minutes for 1.7B-3B models and ~3.8-4.2 hours for 175B-176B models.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model· Method
PaperCodeSource excerpt
“All configurations use AdamW, learning rate 5 × 10−4 , batch size 8, 3 epochs, max input/output lengths 256/128 tokens, and beam search decoding (num_beams=4). LoRA dropout is 0.1; biases are frozen. To assess variance, all configurations are run with three random seeds (42, 123, 456); mean and standard deviation are reported in Section 5.6.” · Show in paper
gpt-oss-120b & gpt-oss-20b Model Card· Method
PaperCodeSource excerpt
“We post-trained the models with quantization of the MoE weights to MXFP4 format[5], where weights are quantized to 4.25 bits per parameter. The MoE weights are responsible for 90+% of the total parameter count, and quantizing these to MXFP4 enables the larger model to fit on a single 80GB GPU and the smaller model to run on systems with as little as 16GB memory.” · Show in paper
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“We use OPT [62] models with 13B, 66B, and 175B parameters and LLaMA [52] with 13B parameters for our evaluation. ... For all of our experiments, we use A2 instances with NVIDIA A100 GPUs on Google Cloud Platform. ... We synthesize workloads based on ShareGPT [51] and Alpaca [50] datasets” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“As shown in the Figure 7, RocketKV achieves up to 3.7× and 3.3× end-to-end speedups on an A100 and H100 GPU, respectively, as well as up to 32.6% peak memory saving.” · Show in paper
Verified
Figure 7 directly shows the ~3.7x A100 speedup, ~3.3x H100 speedup, and ~32.6% peak memory saving at 100K sequence length.
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“(1) Fused reshape and block write. ... (2) Fusing block read and attention. ... (3) Fused block copy.” · Show in paper
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache· Method
PaperCodeSource excerpt
“Specifically, we adopt generation tasks from LM-Eval (Gao et al., 2021) for normal context length evaluation and LongBench (Bai et al., 2023) for long context evaluation, respectively.” · Show in paper
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“As shown in Table 9 and 10, SmoothQuant enables W8A8 quantization of the 530B model at a negligible accuracy loss. The reduced model size allows us to serve the model using half number of the GPUs (16 to 8) at a similar latency, enabling the serving of a >500B model within a single node (8×A100 80GB GPUs).” · Show in paper
Verified
Tables 9 and 10 directly support both the preserved accuracy (73.1% average) and single-node 8-GPU serving at similar latency.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“Although this strategy does not reduce the theoretical amount of compute, it effectively addresses the memory-throughput bottleneck. This provides an order of magnitude speedup for very large models in practice, making it a critical component of our algorithm. ... In combination with mild dampening, the resulting method is robust enough to execute on huge models without issues.” · Show in paper
Verified
The algorithmic formulation in Algorithm 1 and successful execution on 175B models in Table 2 support the robustness and practical efficiency of these two components.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“In Table 4, we show SmoothQuant can quantize all existing open LLMs beyond 100B parameters.” · Show in paper
Verified
Tables 4, 5, 6, and 7 substantiate that SmoothQuant preserves accuracy and perplexity across all tested model families and architectures.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model
PaperCodeSource excerpt
“Our results show that LoRA with r=16 recovers within 11.6 percentage points of full fine-tuning accuracy (59.6% vs. 71.2% exact-match) while training fewer than 1% of parameters and consuming 31% less peak GPU memory.” · Show in paper
Verified
The numbers in Table 2 and the Pareto plots in Figure 1 directly support the accuracy, parameter count, and memory reduction claims for seed 42.
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“LoRA-SVD protects a compact projection subspace in BF16 while retaining full-shape NVFP4 computation, reducing the linear-only loss gap from 1.40% to 0.61%.” · Show in paper
Verified
Table 3 and Section 4.2 directly support the reduction of the linear-only loss gap from 1.40% to 0.61% on 1B/20B-token pretraining.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56× speedup and 2× memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node.” · Show in paper
Verified
The experimental results across accuracy benchmarks (Tables 3–7, 9) and system benchmarks (Figures 8–9, Tables 8, 10) directly support the claimed speedup, memory reduction, model coverage, and single-node 530B serving.
QQQ: Quality Quattuor-Bit Quantization for Large Language Models· Method
PaperCodeSource excerpt
“We conduct the experiments using the LLaMA series (Touvron et al., 2023a,b). We evaluate the model performance on language modeling and zero-shot tasks. For the language modeling aspect, we utilize the WikiText2 (Merity et al., 2016) dataset to measure perplexity. For the zero-shot tasks, we extend the evaluation to a variety of datasets, including PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), and ARC (Clark et al., 2018), utilizing the lm_eval (Gao et al., 2023) toolkit for a comprehensive analysis. Additionally, we select a subset of 128 sequences from the PILE (Gao et al., 2020) dataset to facilitate calibration.” · Show in paper
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“On 3B pretraining with 64B tokens, BF16 and Full-Stack FP4 reach losses of 2.267 and 2.286, a 0.838% gap. Their average zero-shot perplexities are 26.675 and 26.665, respectively, with Full-Stack FP4 averaging 0.10 percentage points lower in accuracy. Native four-block measurements on one RTX 5090 show 2.50–2.83× Root speedups over optimized BF16 and 37.9–42.5% lower AdamW peak memory.” · Show in paper
Verified
All stated loss, perplexity, accuracy, speedup, and memory numbers match the reported experimental results in Sections 4.3, 4.4, Table 4, and Table 5, with the scope of native hardware measurements (four blocks on one RTX 5090) clearly disclosed.
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“Therefore, vLLM features a single KV cache manager within the centralized scheduler, as in Fig. 4. Different GPU workers share the manager, as well as the mapping from logical blocks to physical blocks.” · Show in paper