Search publications
How Inference Compute Shapes Frontier LLM Evaluation· Method
PaperSource excerpt
“We apply three deliberately simple inference-scaling techniques uniformly across every (benchmark, model, condition) cell: 1. Expanded total token budgets of 5M–30M tokens per trajectory (Table 2), sized to task complexity and one to three orders of magnitude above typical benchmark defaults. 2. Context compaction, which replaces earlier turns with a model-generated summary to enable serial scaling beyond the nominal context-window size. We adopt Inspect AI’s default summary-based strategy (UK AI Security Institute, 2024a), triggered when the running context exceeds 130k tokens (about 65% of the smallest 200k context window across models). 3. Iterative resubmission with a hard cap of 999 submissions per trajectory, allowing the model either to refine its previous answer or to try a substantially different approach between submissions. To reduce unproductive loops of near-identical submissions, we also employ a lightweight repetition guard that terminates a trajectory when a separate LLM judge finds three or more consecutive submissions semantically equivalent (Section A.1.3).” · Show in paper
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
PaperCodeSource excerpt
“In contrast, for newer and more complex SE tasks such as ProgramBench and GitTaskBench, complex harnesses can provide greater benefits to stronger models (e.g., Qwen3.7-Max and DeepSeek-V4-Pro).” · Show in paper
Verified
Figures 7-10 clearly show that OpenCode underperforms or matches mini-SWE-agent on weaker models but outperforms it on stronger models in both benchmarks.
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents· Method
PaperCodeSource excerpt
“By combining these factors in a unified three-dimensional experimental matrix, we evaluate each model under each harness across each task, allowing us to compare their individual effects and analyze how they interact in LLM-based SE agents.” · Show in paper
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale· Method
PaperSource excerpt
“The annotation is performed using the gpt-oss-120b model. In addition, for each task we automatically generate auxiliary interfaces extracted from the patches that are explicitly exercised in the test suite.” · Show in paper
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“The model is inconsistent, not incapable. Evaluation is greedy. Yet over the 13 unedited runs of batches 3–15, a single run solves 22–30 tasks, 51 different tasks are solved at least once, and 9 are solved in every run.” · Show in paper
Verified
The observational tallies and independent recomputation support the claimed inconsistency and within-task associations.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?· Method
PaperCodeSource excerpt
“we collect and organize the available information from original sources, such as issue discussions, commit messages, or pull requests, and produce the final task description with two artifacts: (1) a problem statement, which captures the motivation for the change without extending beyond sources, and (2) a list of requirements and optionally interface, which provides the necessary details to fully understand and resolve the issue, grounded in the gold patch and test expectations when applicable.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“On average, prompt-based methods lose more than 50% of their baseline accuracy at longer context windows (≥ 32k), and nearly all prompting methods begin to deteriorate sharply after 8k to 16k tokens.” · Show in paper
Verified
Table 1 shows substantial accuracy drops for all four prompt-only baselines between 0k and 32k-128k.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“Our extensive experiments with an open-weight large language model (LLM) reveal a pronounced gap between high- and low-resource languages, particularly for tasks requiring the model to aggregate multiple facts or predict the absence of information. We also find that, in multilingual settings, LLMs effectively utilize less than 30% of their claimed context length. Although off-the-shelf Retrieval Augmented Generation helps alleviate this to a certain extent, it does not solve the long-context problem.” · Show in paper
Verified
The experimental results in Tables 1, 5, 6 and Figures 5-8 consistently support the claimed gaps across languages, task types, and effective context length, as well as the partial mitigation by RAG.
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
PaperCodeSource excerpt
“(1) The marginal benefit of complex harnesses generally decreases as model capability improves: on SWE-bench Verified, matched-model gains over mini-SWE-agent drop from +9.60/+13.60 pp to +4.07/+5.47 pp, and in Qwen SWE-bench Pro experiments the OpenCode–mini-SWE gap narrows from 6.00 pp to 2.67 pp. (2) This convergence is not uniform across model families. Qwen follows a smooth narrowing trend, whereas DeepSeek shows a non-monotonic pattern, with OpenCode helping more on intermediate models (e.g., DeepSeek-V3.1, DeepSeek-V3.2) and nearly converging with mini-SWE on DeepSeek-V4-Pro (34.3% vs. 33.7%).” · Show in paper
Verified
The reported resolve rates and gaps in Table 3 and Figures 5-6 directly support both the general diminishing-returns trend and the family-specific scaling differences.
Scaling Inference-Time Compute with Teams of Coding Agents
PaperSource excerpt
“Team reward closely tracks the capability of the lead: with identical Luna workers, upgrading the lead from Luna to Terra raises reward by 20.9%, and from Terra to Sol by a further 15.4%, and at every tier the team outperforms the solo agent built on the same model. Cheap workers, however, cannot compensate for a weak lead, as a Terra lead with Luna workers still trails a Sol agent working alone by 10.3%.” · Show in paper
Verified
Figure 7 and Table 3 support the reported percentage differences across lead tiers.
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“Claim tested: the development-set gain of an edit transfers to tasks that played no part in building or selecting it. Neither test confirmed this claim.” · Show in paper
Verified
Both pre-registered tests clearly failed to reach their confirmation criteria, and the authors report the exact confidence intervals, p-values, and power limitations.
Measuring AI Ability to Complete Long Software Tasks· Method
PaperSource excerpt
“Our tasks comprise three distinct task suites: 1. A subset of HCAST [8]: 97 diverse software tasks ranging from 1 minute to 30 hours. 2. RE-Bench [2]: 7 difficult ML research engineering tasks, all eight hours long. 3. Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from 1 second to 30 seconds.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
PaperCodeSource excerpt
“A key contribution of Long-Horizon-Terminal-Bench is dense, subtask-level grading rather than binary pass/fail, which is essential at this difficulty level. Under a strict threshold (R >= 1.0), 10 of the 17 models pass zero tasks (Figure 3); even at R >= 0.95, only 6.4% of runs pass while 30.8% score below 0.05 (Figure 4). The remaining 62.8% of runs make real partial progress, but binary grading would count them as complete failures by giving them the same score as a run that makes no progress.” · Show in paper
Verified
The leaderboard ties at R >= 1.0 and the reward distribution in Figure 4 directly substantiate that dense grading preserves ranking signal and partial-progress information lost under binary grading.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“The finetuned models SWE-Llama 7b and 13b perform surprisingly poorly with BM25 retrieved context. As these models were fine-tuned using the “oracle” retrieval as context, we suspect this shift in context makes it difficult for the model to perform reliably. For instance, SWE-Llama was trained to edit every file included as context whereas in the BM25 setting many files provided in context are not expected to be changed.” · Show in paper
Verified
The performance drop from 3.97% (Oracle) to 0.70% (BM25) for SWE-Llama 13b is documented in Tables 5 and 18, and the explanation aligns with the training setup.
RULER: What's the Real Context Size of Your Long-Context Language Models?· Method
PaperCodeSource excerpt
“Based on these criteria, we develop four NIAH tasks. The "needle" in each of these tasks is a key-value pair inserted into the "haystack" (long distractor texts).” · Show in paper
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues.” · Show in paper
Verified
The experimental results in Tables 2 and 5 support the overarching finding that all evaluated models resolve only a tiny fraction of SWE-bench issues (under 2% for Claude 2, and 3.79% for Claude 3 Opus in the updated Table 5).
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“In this setting, we see increases in performance, with GPT-4 jumping from 1.3% to 3.4% and Claude 2 from 4.8% to 5.9%.” · Show in paper
Verified
Table 6 and Table 18 confirm the performance gains when collapsing unedited context in the oracle setting.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
PaperCodeSource excerpt
“SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. ... In our evaluation of widely used coding models, under a unified scaffold, we observe that their performance on SWE-BENCH PRO remains below 45% (Pass@1).” · Show in paper
Verified
The dataset counts (1,865 problems across 41 repositories) and the <45% Pass@1 performance ceiling (max 43.6% in Table 1 and 17.8% in Table 2) are directly supported by the reported tables and dataset breakdown.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
PaperCodeSource excerpt
“Without the requirements and interface, both models tested (GPT-5 and CLAUDE OPUS 4.1) show significantly degraded performance. In this setting, agents are less constrained and can submit more diverse solutions (particularly for feature additions). Since unit tests expect a narrow set of solutions, verifiers are prone to false negatives, resulting in lower pass rates.” · Show in paper
Verified
Table 3 quantitatively demonstrates the substantial performance drop when requirements and interface specifications are removed.
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K.” · Show in paper
Verified
The results in Table 3 and Tables 10-11 directly support the claim; 10 of 17 models (roughly half) pass the 85.6% threshold at 32K, while nearly all achieve near-perfect scores on vanilla NIAH.