Search publications
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“For every plotted setting, BridgeKV improves over the fine-tuned decode anchor while keeping the same small model for generation. However, the magnitude of the gain depends on the task and model pair. As shown, BridgeKV improves the accuracy of the directly fine-tuned decode anchor by up to 8.4% relative (NarrativeQA, for Qwen2.5-Instruct-7B→Qwen2.5-Instruct-0.5B). In the worst case, the relative improvement is 0.2% (SAMSum, for Llama-3.1-Instruct-8B→Llama-3.2-Instruct-3B)” · Show in paper
Verified
Figure 3 confirms that BridgeKV outperforms the parameter-matched decode anchor in all 30 settings, with relative gains ranging from 0.2% (SAMSum, Llama 8B->3B: 45.3 to 45.4) to 8.4% (NarrativeQA, Qwen 7B->0.5B: 56.8 to 61.5).
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“As shown in Table 3, we observe that for the Llama and Mistral model, KIVI only has up to 2% accuracy drop despite the KV cache being stored in 2bit. ... Since Falcon-7B adopts multi-query attention and only has one head for KV cache, it is already highly compressed compared to Llama-based models. Thus, in Table 3, 4bit KIVI is needed to maintain the accuracy, while 2bit KIVI may have a large accuracy drop in this case.” · Show in paper
Verified
The reported tables and figures accurately reflect these accuracy bounds and the distinction between Llama/Mistral and Falcon-7B.
Resource-Efficient Speculative Decoding for Long-Context LLM Serving
PaperSource excerpt
“Across the three datasets, SpecStream achieves average throughput speedups of 1.41× and 1.32× over TriForce on Qwen3 and InternLM2.5, respectively. The corresponding speedups over SD+Offload are 1.44× and 1.62×.” · Show in paper
Verified
The throughput values plotted in Figure 5 (e.g., Qwen3: 263.4 vs 209.5/229.8 on GSM8K, 17.7 vs 12.2/10.5 on LongBench v2, 16.5 vs 10.8/11.0 on MRCR) support the reported average speedups.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“As shown in Figure 5, we see that as total context length increases, Claude 2’s performance drops considerably; behavior that is also observed in other models. In our evaluation settings, models see a lot of code that may not be directly related to solving the issue at hand, and they seem to frequently struggle with localizing problematic code needing to be updated.” · Show in paper
Verified
Figure 5 and Tables 2-3 directly demonstrate that longer input contexts degrade model resolve rates even when recall of relevant files increases.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction· Method
PaperCodeSource excerpt
“At inference time, we use the learned retention coefficients to rank all cached KV entries globally. For each cached token (ℓ, h, i) present at decoding step t, we assign the retention score G̃ℓ,h,i(t) = ∑T s=t+1 βs−i ℓ,h,i = βt+1−i ℓ,h,i (1 − βT−t ℓ,h,i)/(1 − βℓ,h,i), for βℓ,h,i < 1.” · Show in paper
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“For decode kernels, the performance gap between sparse and dense KV-Cache is negligible (within 1%). For prefill kernels, there is approximately a 10% performance gap.” · Show in paper
Verified
Figure 12 shows decode bandwidth utilization within 1% (83–85%) and prefill TFLOPs/s within ~5–15% (around 10% on average).
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“We show that fixed wall-clock and output-token budgets can materially change apparent agent performance: For example, extending the time limit closes a 20 pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.” · Show in paper
Verified
Figure 6 directly shows the 20.3 pp gap on GPT 5.4 closing completely (74.9% vs 75.3%) and the 29.9 pp gap on GPT 5.4 mini shrinking to 5.3 pp (68.2% vs 62.9%) at 32x timeout.
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“Choosing a separate harness per model on training tasks improves held-out full-credit attainment by 2.53–2.73 percentage points over a single shared harness. Mean-score gains remain within ±0.14 points, and R0.8 gains within 0.31 points. Matching harnesses to models thus matters more for complete delivery” · Show in paper
Verified
Table 10 and the accompanying text in Appendix C.1 confirm that full-credit gains are positive in all 20 repetitions and substantially larger than mean-score or R0.8 gains.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model
PaperCodeSource excerpt
“Expanding from {q,v} at r=8 (53.4%, 344K params) to the full attention set {q,k,v,o} (59.2%, 590K params) recovered 5.8 points at the cost of 71% more trainable parameters and a 6.3% memory increase (1.59 to 1.69 GB). Adding feed-forward layers {q,v,FFN} (58.6%, 786K params) matched {q,k,v,o} in accuracy but required 33% more parameters, an unfavourable trade-off. Notably, LoRA r=16 on {q,v} (59.6%, 590K params) matched {q,k,v,o} at the same parameter budget, suggesting that, in this setting, rank is a more efficient lever than module expansion.” · Show in paper
Verified
Table 2 directly supports the exact accuracy, parameter count, and memory comparisons between module expansion and rank increase.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“Consistent with prior task-adaptation results (Biderman et al., 2024), the strong rank-dependent frontier observed on OSM is not evident in this math setting. In our experiments, it is therefore most pronounced when adaptation installs new factual associations while preserving OOD behavior.” · Show in paper
Verified
Table 3 supports the near-identical math performance and small OOD drops across FFT and QLoRA r=16, 32, and the authors appropriately bound the conclusion to their experimental settings.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Agentic LLM workloads occupy a hardware regime that no prior inference benchmark captures. Across 22 workload profiles, 9 models, and 14 GPU configurations (3,197 sweep rows, 148,077 NCU records), AGENTPERFBENCH identifies a dual pressure that separates agentic serving from chat.” · Show in paper
Verified
The benchmark comparison, saturation sweeps, and roofline/capacity analyses directly support the claim that agentic workloads exhibit this dual-pressure regime uncaptured by prior benchmarks.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“From Table 5, we observe that group sizes 32 and 64 yield similar results, whereas the performance significantly decreases when the group size reaches 128.” · Show in paper
Verified
Tables 5, 6, and 7 directly support the stated sensitivity to group size 128 and the robustness across residual lengths 32 and 128.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model· Method
PaperCodeSource excerpt
“We fine-tune T5-small (60M parameters) for natural-language-to-SQL generation on WikiSQL. Each example is serialized as: translate English to SQL: <Q> table: <H1 | H2 | ···> and the target is the gold human_readable SQL string. We train on 5,000 examples and evaluate on 500 validation examples to maintain a tractable single-GPU sweep.” · Show in paper
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
PaperCodeSource excerpt
“AQLoRA-q averages 4.8 ± 2.0% and is also faster in every session, by 2.1 to 8.1%. Neither result depends on the estimator or the exclusions (Appendix O).” · Show in paper
Verified
Both the 4.8 ± 2.0% speedup and the accuracy parity across all tested models and tasks are supported by the tables.
QQQ: Quality Quattuor-Bit Quantization for Large Language Models
PaperCodeSource excerpt
“Overall, the combination of adaptive smoothing with GPTQ proves to be the most effective method, yielding the lowest perplexity scores. For instance, this approach reduces perplexity by 0.47 in per-group quantization compared with the baseline.” · Show in paper
Verified
Table 3 confirms that B+AS+GPTQ consistently achieves the lowest perplexity across all tested models and granularities, including the 0.47 reduction on per-group LLaMA-3-8B.
Gemma 4 Technical Report
PaperSource excerpt
“Gemma 4 31B is the top open model in the dense category, and both Gemma 4 31B and 26B-A4B show performance equal to much larger open models.” · Show in paper
Verified
Table 4 directly shows Gemma 4 31B as the highest-ranked dense open model and both 31B and 26B-A4B achieving Elo scores comparable to or exceeding much larger open MoE models.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“Only when we choose α from the sweet spot region (0.4-0.6) can we get small quantization errors for both weights and activations, and maintain the model performance after quantization.” · Show in paper
Verified
Figure 10 directly demonstrates the accuracy drop outside the 0.4–0.6 sweet spot for OPT-175B.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models· Method
PaperCodeSource excerpt
“We quantize the inputs and weights of compute-heavy operators like linear layers and BMM in attention layers with INT8, while keeping the activation as FP16 for other lightweight element-wise operations like ReLU, Softmax, and LayerNorm.” · Show in paper
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“The approximation error scales as O(\sigma^2) where \sigma = \max_{i\in\mathcal{E}} |\delta_i|, and a formal bound is given in Appendix A.” · Show in paper
Verified
The derivation in Appendix A.1-A.2 is mathematically sound and establishes both the O(\sigma^2) error bound and O(d^2) storage complexity.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers· Method
PaperCodeSource excerpt
“Concretely, we apply the algorithm to B = 128 columns at a time, keeping updates contained to those columns and the corresponding B × B block of H−1 ... For smaller models, applying dampening, that is adding a small constant λ (we always choose 1% of the average diagonal value) to the diagonal elements of H ... Algorithm 1 Quantize W given inverse Hessian H−1 = (2XX> + λI)−1 and blocksize B.” · Show in paper