Search publications
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“The results show that the compression ratio alone is a poor predictor of end-to-end performance.” · Show in paper
Verified
The joint evaluation across task quality, TTFT, throughput, and realized compression ratio clearly demonstrates that compression ratio alone does not predict end-to-end system or quality outcomes.
Measuring AI Ability to Complete Long Software Tasks· Method
PaperSource excerpt
“Once we have agent success rates and time ratings for each task, we fit time horizons by performing the following logistic regression: p_success(agent,task) = σ((log h_agent − log t_task) · β_agent)” · Show in paper
End-to-End Context Compression at Scale· Method
PaperCodeSource excerpt
“Stage 0: Adapter warmup. We freeze both the encoder and decoder and update only the adapter. • Stage 1: Encoder training. We unfreeze the encoder while keeping the decoder frozen. • Stage 2: End-to-end continual pre-training. We unfreeze the decoder with a small learning rate. • Stage 3: Supervised fine-tuning. We train on the SFT mixture and increase the decoder learning rate.” · Show in paper
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?· Method
PaperSource excerpt
“To evaluate a proposed solution, we apply the generated patch, using unix’s patch program, to the codebase and then execute the unit and system tests associated with the task instance. If the patch applies successfully and all of these tests pass we consider the proposed solution to have successfully resolved the issue. The metric for our benchmark is the percentage of task instances that are resolved.” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“For every plotted setting, BridgeKV improves over the fine-tuned decode anchor while keeping the same small model for generation. However, the magnitude of the gain depends on the task and model pair. As shown, BridgeKV improves the accuracy of the directly fine-tuned decode anchor by up to 8.4% relative (NarrativeQA, for Qwen2.5-Instruct-7B→Qwen2.5-Instruct-0.5B). In the worst case, the relative improvement is 0.2% (SAMSum, for Llama-3.1-Instruct-8B→Llama-3.2-Instruct-3B)” · Show in paper
Verified
Figure 3 confirms that BridgeKV outperforms the parameter-matched decode anchor in all 30 settings, with relative gains ranging from 0.2% (SAMSum, Llama 8B->3B: 45.3 to 45.4) to 8.4% (NarrativeQA, Qwen 7B->0.5B: 56.8 to 61.5).
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?· Method
PaperSource excerpt
“To find high-quality task instances at scale, we use a 3-stage pipeline as follows. Stage I: Repo selection and data scraping. We start by collecting pull requests (PRs) from 12 popular open-source Python repositories on GitHub, producing about ~ 90,000 PRs in total. ... Stage II: Attribute-based filtering. We create candidate tasks by selecting the merged PRs that (1) resolve a GitHub issue and (2) make changes to the test files of the repository... Stage III: Execution-based filtering. For each candidate task, we apply the PR’s test content, and log the associated test results before and after the PR’s other content is applied. We filter out task instances without at least one test where its status changes from a fail to pass (henceforth referred to as fail-to-pass test). We also filter out instances that result in installation or runtime errors.” · Show in paper
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“We identify that a primary source of output degradation is not the residual attention mass on evicted tokens, which existing methods already minimize, but a directional mismatch between the retained and evicted token sets. Specifically, the evicted tokens in practice are often near-orthogonal to the retained ones. Thus, even a small evicted mass could have an oversized impact on the resulting direction distribution and amplify into substantial output error.” · Show in paper
Verified
The mathematical decomposition in Section 3 and empirical measurements in Figure 2 substantiate that directional mismatch drives the output error.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“The per-benchmark results in Appendix B show that degradation is broad on HumanEval, IFEval, MMLU-Redux, and BBH, while TruthfulQA remains comparatively stable.” · Show in paper
Verified
Table 5 clearly shows large drops on four of the five benchmarks and minimal change on TruthfulQA.
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“Figure 3 and Table 6 (Appendix A) show a monotone dose-response for all three edits. The submit rate rises from 0.15 to 0.41; the control gives 0.15. Thinking tokens fall from 659 to 399; the control gives 649. The rate at which the next call makes the first edit falls from 0.66 to 0.49. The four control adapters give 0.66, 0.62, 0.64 and 0.66.” · Show in paper
Verified
The replay results in Figure 3 and Table 6 support monotone dose-response for all three always-on edits while random controls remain near the unedited baseline.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“In summary, our results show that while LLMs are effective at retrieving isolated facts, their capabilities degrade substantially when required to reason over long contexts.” · Show in paper
Verified
Figures 1, 5, 6, and 8 clearly demonstrate the performance gap between retrieval tasks and reasoning tasks across all tested languages.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading· Method
PaperCodeSource excerpt
“A Long-Horizon-Terminal-Bench task follows the Terminal-Bench formulation [22]: it is specified as a Harbor task with (1) a natural-language instruction, (2) a Docker image that defines the terminal environment, (3) a task configuration file, and (4) an oracle implementation or simulator used for grading.” · Show in paper
fadebench
PaperCodeSource excerpt
“With the line the raccoon is fed in 78 of 80 draws, against 48 of 80 without it, and eating while not feeding him falls from 14 of 80 to 0” · Show in paper
Verified
The rates (78/80 vs 48/80; 0/80 vs 14/80) are clearly reported in §5.3, and the authors explicitly note that the comparison is between rates from an earlier run rather than draw-by-draw.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
PaperCodeSource excerpt
“Specifically, our approach involves two complementary measures: (1) exclusively selecting repositories distributed under strong copyleft licenses (GPL) to construct a public set (11 repositories) and a held-out set (12 repositories), and (2) acquiring commercial codebases from real startups to capture enterprise-grade problems in a commercial set (18 repositories).” · Show in paper
Verified
The claim accurately describes the repository selection and access-control strategy implemented by the authors.
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“That is 30.3 against 25.3 over these six batches, higher in all six (Table 3; Figure 4 in Appendix E; sign test p = 0.031).” · Show in paper
Verified
The results in Tables 3 and 10 support the reported means, p-values, and the lack of significance after Holm correction.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“At 64k tokens, the 16x LCLM reaches the first token faster than the smallest uncompressed models (0.21 s, compared with 0.81 s for Llama 3.2 1B and 0.97 s for Qwen3-0.6B) while being far more accurate on LongHealth5 (66.25% vs. 22.50% and 32.75%). The 4x LCLM exceeds the uncompressed Qwen3-4B-Instruct-2507 on LongBench with a 6x lower TTFT.” · Show in paper
Verified
The vLLM-served benchmarks in Figure 5 and Section 6.2 directly substantiate the reported TTFT and accuracy comparisons.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.” · Show in paper
Verified
The reported 400x compression ratio, 3.7x A100 speedup, 32.6% peak memory reduction, and near-Full-KV accuracy are directly supported by the experimental results in Section 4.2, Section 4.3, Figure 5, Figure 7, and Table 2.
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“Table 2 lists all completed pairs: in 9 of 12 only the burst run degraded persistently, in 0 only the no-burst run did, 2 degraded in both arms and 1 in neither (two-sided sign test over discordant pairs, p = 0.00391).” · Show in paper
Verified
The paired design, statistical test, and control experiments cleanly establish history-dependent persistent degradation on vLLM.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“The difference between fake “2bit (K per-channel, V per-token)” quantization and KIVI is that KIVI maintains a full precision key and value cache sliding window for the local relevant tokens. This sliding window is crucial to maintaining accuracy for hard generation tasks such as mathematical reasoning.” · Show in paper
Verified
Table 3 clearly shows that fake 2bit (K-C, V-T) degrades sharply on GSM8K, while KIVI-2 recovers most of the 16bit baseline performance.
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
PaperCodeDataSource excerpt
“We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.” · Show in paper
Verified
The claim is appropriately hedged ('might be inflated') and supported by the empirical divergence in Table 2 between the two DeepSeek-V3 releases on SWE-bench Verified versus their near-identical performance on fresh SWE-rebench tasks.
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“1. Inference scaling is substantial but highly benchmark-dependent. Some benchmarks continue to improve well beyond typical published token budgets – including FrontierMath, TerminalBench, and Humanity’s Last Exam (HLE) – while others show weaker marginal gains under our protocol.” · Show in paper
Verified
Table 3 and Figure 1 directly substantiate both the substantial gains on FrontierMath and HLE and the weak marginal gains on SWE-Bench Pro, TerminalBench, and HealthBench under the tested protocol.