Search publications
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“Table 3 compares the training cost of the KV-stitching module with that of the parameter-matched PEFT baseline, which fine-tunes the decode anchor alone with LoRA at the same number of trainable parameters as the stitched model. Training the module is cheaper in every setting.” · Show in paper
Verified
In all 16 rows of Table 3, the sum of Stage 1 and Stage 2 GPU-hours is strictly less than the PEFT baseline GPU-hours.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“For moderate levels of parallel generation (4 ≤ n ≤ 32), FlashInfer’s composable formats yield consistent speedups for both ITL and TTFT. Peak speedups occur at n = 4, where ITL decreases by 13.73% for the 8B model and 17.42% for the 70B model, while TTFT is reduced by 16.41% for the 8B model and 22.86% for the 70B model.” · Show in paper
Verified
Figure 10 confirms the exact percentage reductions at n=4 and consistent speedups for 4 <= n <= 32.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier· Method
PaperCodeSource excerpt
“We use Qwen3-4B (Yang et al., 2025a) as the base model and compare full fine-tuning (FFT) with QLoRA adapters of rank r ∈ {8, 16, 32, 64}. Each training example consists of a question and its gold answer, with the autoregressive loss applied only to answer tokens. All runs are repeated over five random seeds.” · Show in paper
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“LoRA-SVD reduces the error of outputs, input gradients, and weight gradients throughout the observed training trajectory. In a separate paired-forward check on real text, quantized and unquantized forwards from the same checkpoint and input show 29.8–33.0% lower final-output MSE with LoRA-SVD across d = 512–2048.” · Show in paper
Verified
Figure 2 visually supports the ~10% relative MSE reduction across the 1B trajectory, and Section 3.2 reports the 29.8–33.0% final-output MSE reduction.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“At ≈ 2.2 bit (group-size 128; using FP16 scale and 2-bit zero point per group) the perplexity increase is already less than 1.5 points, while dropping to 0.6 - 0.7 at ≈ 2.6 bit (group-size 32), which is only slightly worse than vanilla 3-bit and might be interesting for practical kernel implementations. Further, if we reduce group size to 8, we can apply ternary (-1, 0, +1) quantization, which achieves 9.20 WikiText2 PPL on OPT-175B, a less than 1 point drop.” · Show in paper
Verified
Table 7 directly supports the 2-bit grouped perplexity numbers; the ternary group-size 8 result (9.20 PPL) is reported in the text.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“At 64k tokens, the 16x LCLM reaches the first token faster than the smallest uncompressed models (0.21 s, compared with 0.81 s for Llama 3.2 1B and 0.97 s for Qwen3-0.6B) while being far more accurate on LongHealth5 (66.25% vs. 22.50% and 32.75%). The 4x LCLM exceeds the uncompressed Qwen3-4B-Instruct-2507 on LongBench with a 6x lower TTFT.” · Show in paper
Verified
The vLLM-served benchmarks in Figure 5 and Section 6.2 directly substantiate the reported TTFT and accuracy comparisons.
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“For the second moment, direct NVFP4 gives 0.0116, whereas the complete pipeline gives 0.0053, a 54.3% reduction. ... In separate 1B native-state ablations, retaining NVFP4 state iteration while removing any required preprocessing stage caused rapid instability or divergence, so no comparable endpoint loss existed. To our knowledge, this is the first demonstration of stable NVFP4 AdamW momentum-state storage in an integrated 1B pretraining run.” · Show in paper
Verified
The reconstruction error reductions, bit-width accounting, and 1B/20B-token pretraining stability are all directly supported by the reported data.
QLoRA: Efficient Finetuning of Quantized LLMs· Method
PaperCodeSource excerpt
“We use this feature to allocate paged memory for the optimizer states which are then automatically evicted to CPU RAM when the GPU runs out-of-memory and paged back into GPU memory when the memory is needed in the optimizer update step.” · Show in paper
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“In Figure 12 (a), we see that increasing the size of the decoder is much more beneficial in terms of pre-training loss than increasing the size of the encoder.” · Show in paper
Verified
Figure 12(a) and Table 7 directly support both the pre-training loss reduction and the mixed downstream results.
Resource-Efficient Speculative Decoding for Long-Context LLM Serving· Method
PaperSource excerpt
“We implement SpecStream on SGLang 0.5.17 [21] and evaluate two Target/Draft model pairs, Qwen3-32B/0.6B [14] and InternLM2.5-20B-Chat/1.8B-Chat [15]. The main experiments use two NVIDIA A800 80GB PCIe Gen4 GPUs and 240GB of host memory.” · Show in paper
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6× less peak memory (including model weight). This reduction in memory usage enables up to 4× larger batch size, bringing 2.35× ∼ 3.47× throughput on real LLM inference workload.” · Show in paper
Unsupported
The headline claim overgeneralizes: Section 4.2.2 explicitly notes that Falcon-7B needs 4bit KIVI to maintain accuracy and suffers a large accuracy drop with 2bit KIVI, and the 2.6x memory reduction and throughput gains were only measured on Llama-2-7B.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“We find that in some settings SWE-Llama 13b is competitive with Claude 2 and is capable of processing contexts exceeding 100,000 tokens.” · Show in paper
Verified
In the Oracle retrieval setting (Table 18), SWE-Llama 13b achieves 3.97% resolved vs. Claude 2's 4.80%, and Table 4 confirms its >=100,000 token context capacity.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“Comprehensive kernel-level and end-to-end evaluations demonstrate FlashInfer’s ability to significantly boost kernel performance across diverse inference scenarios: compared to state-of-the-art LLM serving solutions, FlashInfer achieve 29-69% inter-token-latency reduction compared to compiler backends for LLM serving benchmark, 28-30% latency reduction for long-context inference, and 13-17% speedup for LLM serving with parallel generation.” · Show in paper
Verified
All three headline percentage ranges match the empirical results reported in Sections 4.1, 4.3, and 4.4.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction· Method
PaperCodeSource excerpt
“We follow the same training procedure as [Bui et al., 2025] described in Section 2.2 but replace the local capacity constraint by a global counterpart Lcap = ∑T t=1 max(0, ∑ℓ,h ∑t i=1 βt−i ℓ,h,i − Mglobal), where Mglobal is the target global KV budget. We only train retention gates and freeze LLM weights.” · Show in paper
Resource-Efficient Speculative Decoding for Long-Context LLM Serving
PaperSource excerpt
“Across the two model pairs, SpecStream achieves an average throughput speedup of 1.036× over SPECTRE+Offload. Because SPECTRE uses one additional GPU, normalizing by GPU count gives SpecStream an average improvement of 55.4% in output throughput per GPU.” · Show in paper
Verified
Figure 10 and the GPU count normalization (3 GPUs vs. 2 GPUs, 1.036 * 3/2 ≈ 1.554) directly support the 55.4% per-GPU throughput improvement.
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
PaperCodeSource excerpt
“Figure 6 confirms substantial gains over FP16: Qwen2.5-7B achieves 2.0×–2.5× speedup (PRO 6000) and Llama 3.1-8B reaches 3.5× (RTX 5090), while memory usage drops by 1.5×–2.8×. Crucially, residual overhead is marginal: compared to uncompensated NVFP4, latency increases by only 3%–9%.” · Show in paper
Verified
The latency and memory numbers in Figure 6, Figure 8, and Table 10 support the reported speedup, memory reduction, and 3%–9% overhead over uncompensated NVFP4.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“For each span and head, we factorize the pre-RoPE key matrix and the value matrix with a truncated SVD up to rank , and a lightweight predictor assigns each component a gate from the singular spectrum and the span summary: ” · Show in paper
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“In vLLM, we adopt the first-come-first-serve (FCFS) scheduling policy for all requests, ensuring fairness and preventing starvation. ... we implement an all-or-nothing eviction policy, i.e., either evict all or none of the blocks of a sequence. Furthermore, multiple sequences within one request (e.g., beam candidates in one beam search request) are gang-scheduled as a sequence group.” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation· Method
PaperSource excerpt
“The first stage trains only ψ, with both anchors frozen, and combines two losses: an alignment loss that keeps the stitched cache close to the native cache of Md, and a distillation loss that makes Md, reading the stitched cache, match the next-token distribution of Mp.” · Show in paper
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“Overall, these findings suggest that frontier capability cannot be fully characterized by a single benchmark score measured under a single inference-time protocol. Observed performance depends not only on the model, but also on how much inference-time compute it is given and how that compute is allocated.” · Show in paper
Verified
The claim is well-supported by the multi-benchmark experiments showing that token budgets, repeated submissions, feedback, and parallel vs. serial allocation all substantially affect measured performance.