Search publications
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
PaperSource excerpt
“Splitting the variance of solved-or-failed over the 642 tasks every combination attempted, task identity accounts for 31.7%, the model for 2.65%, the scaffold for 0.29% and the model–scaffold interaction for 0.36%; the residual 65.0% is task-by-run variation.” · Show in paper
Verified
The numbers in Appendix B and Section 4.5 are consistent.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces· Method
PaperSource excerpt
“Terminus 2 has a single tool, a headless terminal, and completes tasks using only Bash commands.” · Show in paper
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“Choosing a separate harness per model on training tasks improves held-out full-credit attainment by 2.53–2.73 percentage points over a single shared harness. Mean-score gains remain within ±0.14 points, and R0.8 gains within 0.31 points. Matching harnesses to models thus matters more for complete delivery” · Show in paper
Verified
Table 10 and the accompanying text in Appendix C.1 confirm that full-credit gains are positive in all 20 repetitions and substantially larger than mean-score or R0.8 gains.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2–3× compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3× and declines by only 0.90 and 2.40 pp at 4× and 5×, respectively.” · Show in paper
Verified
The experimental results in Table 1, Tables 4-7, and Table 9 support the 2-3x near-lossless claim across the tested models and tasks, noting that LLaMA-3.1-8B and Qwen2.5-14B used FlashJoLT.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.” · Show in paper
Verified
The experimental results across Tables 1, 2, 3, 7 and Figure 5 directly support that DBTrimKV reduces KV memory while matching or surpassing full-cache inference across all tested benchmark categories.
Scaling Inference-Time Compute with Teams of Coding Agents
PaperSource excerpt
“Our results show that teams outperform solo agents, but at a steep price: a team composed of all strong agents raises reward over a single agent working alone, yet costs 3 to 7 times as much in inference dollars and, in most configurations, takes longer to finish (Section 4).” · Show in paper
Verified
Figure 2, Section 4, Table 3, and Table 4 substantiate the reward gains, 3–7× cost increase, and 1.4×–3.8× wall-clock time increase on ProgramBench.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
PaperCodeSource excerpt
“We evaluate 17 frontier models and find that agents consume on average 9.8M tokens per task, with roughly 239 episodes and 88.9 minutes of execution time per run, making Long-Horizon-Terminal-Bench substantially more demanding than prior terminal-based benchmarks. Even the strongest tested model, Grok 4.5, achieves only 28.3% pass@1 at a partial-reward threshold of 0.95 and 19.6% at a perfect-reward threshold of 1.0, while the mean pass rate across models is just 6.4% and 3.2% under the two thresholds, respectively.” · Show in paper
Verified
The reported pass rates, token counts, episode counts, and execution times are directly supported by the quantitative results in Table 1, Figure 3, and Section 3.1.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading· Method
PaperCodeSource excerpt
“We evaluate models using the Harbor framework with the Terminus-2 agent harness [10], which interacts with each task environment through a single long-horizon terminal session. Additionally, for GPT-5.3, we instead use Codex [28] as the agent harness.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio.” · Show in paper
Verified
Both Table 2 and Table 4 clearly demonstrate extreme workload sensitivity in CaM's accuracy and realized compression ratio.
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“Our additional ablations demonstrate that training on longer sequences does not always lead to better performance on RULER” · Show in paper
Verified
Both the LWM ablation in Figure 4 and the main results in Table 3 support that longer training context does not guarantee superior RULER performance.
Scaling Inference-Time Compute with Teams of Coding Agents
PaperSource excerpt
“Cheap workers pay off only when directed toward planning (Figure 6). Relative to PIV, planning raises reward by 5.6% while lowering cost by 22.2% under blocking, and by 4.0% while lowering cost by 30.9% under asynchronous push. Directed toward implementation, by contrast, Luna workers reduce the team’s reward below that of the Sol solo agent while increasing its cost” · Show in paper
Verified
Figures 6 and 7, along with Table 3 and Table 4, support the stage-dependent payoff of cheap Luna workers.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“Across 19 agents, rankings remain highly correlated between TB2.0 and TB2.1 (Spearman ρ = 0.958), while individual scores shift by up to 12 percentage points and leaderboard positions move by up to three ranks.” · Show in paper
Verified
The correlation, confidence interval, and rank/score shifts are directly supported by Figure 4 and the 19-agent evaluation in Section 6.1.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark· Method
PaperSource excerpt
“GLM-5.3 (thinking on), Kimi K3 (reasoning effort max) and Inkling (xhigh) ran with a 1M-token context window and no cap on thinking tokens or turns; the only cap besides the 50-minute budget is mini-swe-agent’s default 30 s per-command timeout, which is part of that scaffold.” · Show in paper
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“Figure 4 (middle-right) shows that the 34B model is significantly better than the 6B model on RULER for both performance at length of 4K and the relative degradation, suggesting the benefit of scaling model sizes for better long-context modeling.” · Show in paper
Verified
Figure 4 (middle-right) clearly shows Yi-34B-200K outperforming Yi-9B-200K and Yi-6B-200K across all tested lengths.
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
PaperSource excerpt
“First-attempt success varies widely: Claude ranges from 96% in JavaScript and 86% in Python down to 36% in C++ and only 18% in Java, Codex spans 44% (C++) to 88% (Python and JavaScript), and Gemini runs from 32% (Python) to 84% (JavaScript). After repair, nearly every configuration converges: Claude reaches 100% in all four ecosystems, Codex falls short only in C++ (94%), and Gemini falls slightly short in Java (96%) and C++ (98%).” · Show in paper
Verified
Figures 4-6 and Section 6.1 directly report the exact first-attempt and post-repair convergence rates across all 600 primary instances.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks· Method
PaperSource excerpt
“For each task, the evidence agent produces a structured report containing information on the benchmark, the task specification, trace information, and community reports. ... An auditor applies the rubric to each task. ... A verifier reviews the evidence and the auditor’s label.” · Show in paper
Scaling Inference-Time Compute with Teams of Coding Agents
PaperSource excerpt
“Greater communication, however, does not yield higher reward. Sol and Terra teams are unchanged within noise, and Luna teams lose 4.4–7.1 points (Figure 9).” · Show in paper
Verified
Figure 9, Appendix C.10, and Table 6 directly support the null/negative effect of inter-agent messaging on reward.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks· Method
PaperSource excerpt
“We evaluate 19 agents from multiple providers and compare the accuracy in both TB 2.0 and TB 2.1, as illustrated in Fig. 4. The correlation is high (Spearman ρ = 0.958), with an OLS fit of TB2.1 = 2.1 + 1.04 · TB2.0.” · Show in paper
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
PaperCodeSource excerpt
“There is also a significant performance gap between the public and commercial set, where the best models score less than 20% in the commercial set, highlighting the difficulty of navigating enterprise codebases.” · Show in paper
Verified
Tables 1 and 2 directly show that all tested models score below 20% on the commercial set and significantly higher on the public set.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks· Method
PaperCodeDataSource excerpt
“We conduct all our experiments using the open-weight Llama 3.1-Instruct model, which is an instruction-finetuned version of Llama 3.1 (Meta AI 2024). ... For our RAG experiments, we make use of an off-the-shelf and a specialised multilingual models – Jina-reranker (JinaAI) (Multilingual 2024) and paraphrase-multilingual-mpnet-base-v2 (MPNet) (Reimers and Gurevych 2020), respectively.” · Show in paper