Search publications
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“To this end, we introduce SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.” · Show in paper
Verified
The dataset construction pipeline, repository breakdown (Figure 3, Table 10), and task statistics (Table 1) substantiate the existence, size, and automated collection of the 2,294-instance benchmark.
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“For each benchmark, later model generations unlock a larger proportion of tasks. This is indicated by a positive, significant main effect of model generation on the probability of unlocking a task on all six benchmarks (Table 4), with the largest effect on FrontierMath and Cyber CTFs.” · Show in paper
Verified
All coefficients and significance levels in Table 4 match the stated claim.
fadebench
PaperCodeSource excerpt
“Drift-act traces take two main shapes, on the results lane's codes. Either the cost is named and argued past (158 of 185, mostly with the world stated correctly), or it is never mentioned (27, mostly with a fact of the scene stated wrongly).” · Show in paper
Verified
The counts (158/185, 27/185, 0/185, and 4/185 on blind re-rate) match §5.6, and the authors explicitly note that the blind second rater is from the same model family.
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows· Method
PaperCodeDataSource excerpt
“Runs use a 200K-token input-context threshold, beyond which each harness applies its default context management, and a 150-turn interaction limit. ... Each task–model–harness combination is evaluated once.” · Show in paper
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
PaperCodeSource excerpt
“On average, the reference solutions span 107.4 lines of code across 4.1 files. Every problem involves at least 10 lines of change, and over 100 tasks demand more than 100 lines of modification.” · Show in paper
Verified
The reported averages and distributions in Figures 1 and 2 support the claim of substantial multi-file modifications.
BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence
PaperSource excerpt
“However, Table 3 shows that, in our long multimodal document QA setting requiring multi-hop evidence, Colpali-RAG substantially degrades end-to-end answer quality even when Gemini is used as the downstream generator.” · Show in paper
Verified
Table 3 directly supports the substantial degradation in Audit, Accuracy, and Fidelity for Colpali-RAG relative to all three Gemini-only baselines.
Scaling Inference-Time Compute with Teams of Coding Agents· Method
PaperSource excerpt
“Each stage-specific guidance instructs the lead to complete the full workflow while using workers only in the designated phase. This design isolates the value of delegation at each phase. We then compose all three interventions into PIV, our heavy-guidance variant, to evaluate their joint effect when workers are used throughout the workflow” · Show in paper
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“3.2. Generational gains are driven mainly by reach and reliability rather than efficiency” · Show in paper
Verified
The regression results in Table 4 and curve summaries in Table 9 support that reach and reliability improve more consistently across generations than token efficiency.
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
PaperCodeSource excerpt
“The full NanoHarness configuration achieves the strongest performance among variants, improving over mini-SWE-agent on both models and outperforming any single component. Although NanoHarness still slightly trails product-level harnesses such as OpenCode and Claude Code, it closes most of the gap with a lightweight modular design, suggesting that a compact set of representative components can reproduce much of the benefit of complex harnesses.” · Show in paper
Verified
Table 6 and Figures 11-12 confirm that full NanoHarness outperforms all single-component variants and closes the gap to OpenCode and Claude Code to within 1.13–2.08 percentage points.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Adding a shared 1024-token synthetic prefix brings deep-turn E2EL MAPE to 3.2% and aggregate E2EL MAPE to 2.4%, within the observed noise scale. Hence synthetic distributional profiles in this paper use this prefix-aware configuration.” · Show in paper
Verified
Table 11 directly reports the ablation results confirming 2.4% aggregate and 3.2% deep-turn E2EL MAPE.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
PaperSource excerpt
“In general, there is a positive correlation (r = 0.436, p < 0.001) between human-predicted and empirical difficulty, and 93.3% of human-hard tasks are also empirically hard (Figure 7).” · Show in paper
Verified
Figure 7 and the accompanying text in Section 4.3 directly support the correlation and percentage breakdowns.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Corollary 3.2 provides an intuition for why eviction can improve long-context behavior. Its condition, , only requires useful tokens to be retained at a higher logit-weighted average rate than distractors. Thus, any retention rule that suppresses distractor mass more than useful-token mass reduces distractor-induced dilution.” · Show in paper
Verified
The proofs of Proposition 3.1 and Corollary 3.2 in Appendix A.1 are mathematically sound, and Figure 1 provides consistent empirical support.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“As shown in Figure 6, our agent substantially improves performance over the same LCLM with 16x compressed context and no expansion.” · Show in paper
Verified
Table 36 and Figure 6 provide exact per-task and average accuracy numbers showing large gains (+18.50, +23.19, and +25.50 points at 4k, 8k, and 16k).
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading· Method
PaperCodeSource excerpt
“For each problem, we build a complete but deliberately broken terminal-only project. ... Each task includes reproducible public asset-generation scripts, a weak baseline implementation, an official gold solution, a multi-step solve.sh, and a hidden verifier.” · Show in paper
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“The pooled edit exceeds the unedited model by 2.7 tasks over 12 batches (95% CI +0.3 to +5.0, p = 0.03; higher in 9 batches, sign test p = 0.15). It exceeds the mean of the random controls by 3.6 tasks in the 7 batches that contain both (CI +2.2 to +5.1, p < 0.001; higher in 7 of 7, sign test p = 0.016).” · Show in paper
Verified
The numbers in Table 2 support the claim, and the authors appropriately qualify that the 0.125–0.5 pool is post hoc and the development tasks are in-sample.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs· Method
PaperSource excerpt
“Trace replay profiles preserve the exact recorded requests from the source traces. Distributional synthetic profiles sample multi-turn requests from per-turn ISL, OSL, and turn-count distributions fitted to those traces, producing representative workloads that can be reissued across models, hardware, backends, and concurrency settings, without the runtime cost of full replay.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“On GSM8K [41] with 8-shot chain-of-thought prompting [42] and exact-match scoring (n=1319 per cell), JoLT is near-lossless at 2× and 3× on both Mistral and LLaMA-2 (Table 7 in Appendix C). All free-zone deltas fall within their 95% confidence intervals of zero, with the largest shift at −1.21 pp (Mistral 3×).” · Show in paper
Verified
Tables 7 and 10 support the claims with the stated caveats (single model and single seed for LongBench).
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“The longer agentic profiles exceed the single-H100 HBM budget: coding agent exceeds the wall by C = 40 and reaches 155 GB at C = 80, while terminal reaches 126 GB (Appendix Figure 4b clips both at the 80 GB wall).” · Show in paper
Verified
The analytical CF model in Figure 4b and Table 14b directly supports the capacity saturation thresholds.
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents· Method
PaperCodeDataSource excerpt
“We use the Qwen2.5-72B-Instruct model [Qwen et al., 2025] (prompt in Appendix B.2) to generate up to three candidate JSON recipes per task. If an error occurs during the subsequent scripted installation or test execution (derived from a recipe), the LLM attempts to refine that recipe by analyzing error logs and the original instructions (see correction prompt in Appendix B.4).” · Show in paper
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
PaperCodeDataSource excerpt
“For Task Complexity (where 'high-score' implies >1 hour to solve; 100 high-score vs. 313 low-score examples in validation), our fine-tuned model achieved 81% accuracy and a weighted F1-score of 0.82. This is an improvement over the baseline Qwen-72B-Instruct, which achieved 68% accuracy. For Test Patch Correctness (180 high-score vs. 233 low-score examples), the model achieved 67% accuracy (weighted F1: 0.65). For Issue Clarity (84 high-score vs. 329 low-score examples), it achieved 79% accuracy (weighted F1: 0.76).” · Show in paper
Verified
All reported accuracy and weighted F1 numbers match Table 4 exactly. Note that on Issue Clarity, the vanilla model achieved 80% accuracy (0.77 weighted F1) vs 79% (0.76) for fine-tuned, which is visible in Table 4.