Search publications
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“SnapKV achieves the best throughput, on par with All-KV on Llama and even surpassing All-KV on Mistral.” · Show in paper
Verified
Figure 3 directly supports that SnapKV-0.75 achieves the highest average throughput among all evaluated methods on both models.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces· Method
PaperSource excerpt
“The result is a simplified taxonomy with three high-level classes – Execution, Coherence, and Verification —- each with subclasses.” · Show in paper
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
PaperSource excerpt
“We find that command error rates range from 9.2% (Grok 4) to 26.7% (GPT-OSS-120B).” · Show in paper
Verified
Figure 9 and the text on pages 9-10 support the reported error rate range and failure category percentages.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“The largest gains are concentrated among tasks with instruction–verification mismatches. In TB 2.1, five tasks increased from pass rates of at most 15% to over 50% after repairs centered on the verifier, accounting for roughly 62.7% of the total absolute pass-rate change on the modified-task subset (full breakdown in Appx. C).” · Show in paper
Verified
Table 7 confirms the five tasks increasing from <=15% to >50%, all of which are listed under instruction-verification mismatch in Table 2.
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“Harness choice substantially affects performance within individual models. General-task score gaps across harnesses range from 5.0 to 9.5 points within nine models, and reach 39.2 for Qwen (Table 2). ... Multimodal gaps are larger for several models: Kimi ranges from 28.1 with Codex to 72.7 with OpenClaw, with preferred harnesses differing across backends (Table 3).” · Show in paper
Verified
The reported ranges match the values in Tables 2 and 3.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“The base LLM stays frozen, and the router adds under 0.1% parameters.” · Show in paper
Verified
Table 24 confirms parameter overhead is 0.054%–0.086% (<0.1%), and Table 3 confirms the 2.8–3.8 point drop when using training-free routing.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark· Method
PaperSource excerpt
“Under the lock, the sandbox can reach only the model endpoint during the agent phase (the allowlist names that hostname alone), web tools that scaffolds execute outside the sandbox are disabled, and the images carry no fixing commit in their git history.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“There is an extraction of a consistent tuple F′ of deterministic length ℓ′ = Θ(k), still in one of the two stream orientations, and a new discrete context C′ for which H(C′) ≤ Λ′ ≤ CqP log(w + 2)(σ + wP) + Cwσ, |domain of each slot of F′| ≤ Cq^d , Δ′ ≤ CK∗, 1 ≤ w ≤ Cσ^{1+η/2}/D.” · Show in paper
Verified
The standalone nature of the new context C' and the uniformity of all constants across admissible histories are carefully verified in Section 6.5 and the proof of Lemma 6.1 on page 47.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Dropping the residual entirely (rank-only compression) raises 2× perplexity from 6.54 to 7.27, a gap that additional rank cannot close.” · Show in paper
Verified
Table 11 directly supports both the necessity of the rotated residual and the saturation at 4 bits.
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“We evaluate the effective context length for two models with non-Transformer architectures: RWKV-v5 (Peng et al., 2023) and Mamba-2.8B-slimpj (Gu & Dao, 2023). We find that both models demonstrate significant degradation when extending context size to 8K, and both underperform the Transformer baseline Llama2-7B by large margins up till the length of 4K, beyond which Llama2 shows poor length extrapolation performance (Figure 4 right).” · Show in paper
Verified
Figure 4 (right) directly supports the performance gap up to 4K and the degradation at 8K, though Mamba-2.8B is smaller in parameter count (2.8B vs 7B).
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
PaperCodeSource excerpt
“Among the models tied at one solved task at R >= 0.95, mean reward spans from 0.27 for GPT-5.4 to 0.32 for GLM 5.2. Pass rate and mean reward are positively but only moderately correlated (Spearman \rho = 0.74), and they can produce different model rankings because they measure different aspects of long-horizon performance: pass rate captures full task completion, whereas mean reward captures sustained partial progress.” · Show in paper
Verified
Figure 3 confirms the exact mean reward values for the four models tied at one solved task at R >= 0.95.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
PaperSource excerpt
“Execution errors dominate for Opus 4.5 and GPT-5.2, while coherence and verification errors occur at lower rates. Conversely, the open sourced model evaluated (Qwen Coder) displays a more balanced error pattern, with higher errors across all failure modes.” · Show in paper
Verified
Figure 8 clearly illustrates higher execution error bars relative to coherence and verification for Opus 4.5 and GPT-5.2, and elevated, more uniform bars across all three categories for Qwen Coder 480B.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark· Method
PaperSource excerpt
“Figure 2 traces the tasks through three repair rounds, an engineer review and a final review. Round 1 repaired 40 container images whose reference patches carried test and fixture files or whose instructions lacked the standard template. Round 2 ran an automated audit of difficulty calibration, instruction–test alignment and reward hacking; it flagged 387 tasks (53%), found 326 clean and errored on 18.” · Show in paper
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“We evaluate seven harnesses with ten backends on general tasks and seven on multimodal tasks, using paired comparisons on shared tasks. Our evaluation reveals substantial, model-dependent performance differences: no harness performs best across all ten backends, and GLM-5.3’s general-task full-credit rate varies by 13.5 percentage points across harnesses.” · Show in paper
Verified
The full-suite tables (Tables 2 and 3) directly show that no single harness leads across all ten backends on general tasks or all seven backends on multimodal tasks.
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale· Method
PaperSource excerpt
“To bypass GitHub API rate limits and enable large-scale processing, we clone repositories in distributed map-reduce jobs and extract pull request patches directly from local git histories, avoiding per-instance API requests.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“Segmentation: fixed-length chunks of the same mean length cost 2.4 to 2.9 points and dropping anchors 3.0 to 3.9, so where spans begin and which tokens stay exact both matter.” · Show in paper
Verified
Table 3 directly supports the stated point drops.
Scaling Inference-Time Compute with Teams of Coding Agents
PaperSource excerpt
“Planning workers improve reward at every team size: two workers already improve on the solo agent by 3.2%, and sixteen raise the gain to 7.1%. Implementation and verification workers, in contrast, improve from two to four workers but then plateau, with their reward changing by −0.015 and −0.011, respectively, when scaling to 16 workers.” · Show in paper
Verified
Figure 8 and Appendix C.8 (Figure 24) substantiate both the scaling behavior and the underlying mechanism.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“All-KV was not the consistent winner in terms of accuracy across the various workloads in our benchmark. Instead, under most workloads, one of the compression techniques achieved even better results.” · Show in paper
Verified
Table 2 directly shows compressed methods outperforming All-KV on most model-dataset combinations.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?· Method
PaperSource excerpt
“We follow our data collection procedure and collect 19,000 issue-PR pairs from an additional 37 popular Python package repositories. In contrast to Section 2.1, we do not require that pull requests contribute test changes. ... To eliminate the risk of data contamination, the set of repositories in the training data is disjoint from those included in the evaluation benchmark. ... For memory efficiency, we fine-tune only the weights of the attention sublayer using LoRA Hu et al. (2022), and exclude training sequences with more than 30,000 tokens, reducing the effective size of the training corpus to 10,000 instances.” · Show in paper
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“As shown in Table 5, using the token contextual embedding xt as input performs better, particularly under tight budgets, suggesting that xt provides a stronger long-horizon utility signal than the projected KV representations alone.” · Show in paper
Verified
Table 5 directly supports the claim.