Search publications
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?· Method
PaperCodeSource excerpt
“We conduct an LLM-as-a-judge analysis for failure modes of different models, utilizing GPT-5 as the judge. ... For each of the models in Table 4, we programmatically filter to only unresolved instances of SWE-BENCH PRO and collect the last 20 turns of each rollout. ... With a system prompt providing strict descriptions of the failure buckets and overall SWE-Agent format, we feed the trajectory input and prompt the GPT-5 judge to first produce a 1-paragraph reasoning and then an ultimate selection of one failure mode per instance.” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression· Method
PaperCodeSource excerpt
“As shown in Figure 3, RocketKV performs coarse-grain KV cache eviction in the first stage. The purpose of this stage is to remove KV tokens with low importance while keeping the majority of important tokens. In the second stage, it conducts fine-grain dynamic KV token selection on the remaining KV tokens, followed by top-k sparse attention.” · Show in paper
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents· Method
PaperCodeDataSource excerpt
“We fine-tune Qwen 2.5-72B-Instruct using annotations from SWE-bench Verified. For each of the over 3,800 examples, the model receives the issue description, the canonical solution patch, and the test patch as input. It is then prompted to predict one of three binary quality labels: Issue Clarity, Task Complexity, or Test Patch Correctness.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
PaperCodeSource excerpt
“The average costs span from about $3.6 to $26 per task. Although Grok 4.5 achieves the highest pass rate, it does so at only about $11 per task, well below the most expensive models. GPT-5.6-sol and GPT-5.5 form a second tier of strong but substantially pricier models at roughly $21 per task, and GPT-5.4 is the most expensive overall (about $26 per task) with a much lower pass rate.” · Show in paper
Verified
Table 1 and Figure 5 directly support the cost-efficiency comparison and the conclusion that higher spending does not guarantee higher pass rates.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition–retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime.” · Show in paper
Verified
The five-seed results in Figure 1 and Table 5 directly support the full claim, including the relative positions of FFT and QLoRA ranks 8, 16, 32, and 64.
How Inference Compute Shapes Frontier LLM Evaluation· Method
PaperSource excerpt
“For total budget B, we estimate pass@k by splitting B across k independent trajectories of B/k tokens each and counting a task as solved if any trajectory’s first correct submission falls within budget. pass@k is computed task-wise using the unbiased estimator of Chen et al. (2021).” · Show in paper
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale· Method
PaperSource excerpt
“The pipeline has five stages: 1. Preliminary Data Collection: Mining and filtering candidate PRs from global GitHub activity. 2. Setup Synthesis: Deploying an interactive agent to autonomously infer repository-level installation and testing scripts. 3. Execution-based Validation: Validating environments through dual-pass execution (pre- and post-fix) to extract test oracles. 4. Filtering By Issue Clarity: Removing underspecified tasks and excluding instances with potentially sensitive information. 5. Metadata Enrichment: Tagging instances with diagnostic features to enable flexible selection.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“SEKV gains more from extra budget than ShadowKV, so its gap to ShadowKV narrows from 3.4 points at 5% to 1.0 at 20% on RULER and from 2.8 to 0.3 on NIAH, and on LLAMA-3.1-8B SEKV matches ShadowKV on NIAH at 20%.” · Show in paper
Verified
Table 4 directly supports all stated numbers.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“Presented in the second row of Figure 5, RocketKV achieves near the Full-KV accuracy across all models, even with a token budget of 256. In fact, it reaches 100% accuracy on Llama3.1-8B-Ins under 256 token budget, which corresponds to a compression ratio of over 400× with a maximum sequence length of 109K tokens. In contrast, all other methods suffer from substantial accuracy drops.” · Show in paper
Verified
The NIAH scores in Figure 5(d-f), Table 2, Table 10, and Table 16 directly support the claim.
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software· Method
PaperSource excerpt
“Our agent evaluation protocol divides this agent behavior into two skills, a generation skill, in which it produces code and a dependency manifest for a given task, and a repair skill, in which it iteratively revises both specified code and dependency manifest in response to runtime failures.” · Show in paper
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?· Method
PaperCodeSource excerpt
“We use the SWE-Agent [16] scaffold. ... For open-source LLMs, we use vllm to host each model. Models are hosted on a single node, with 8 H100 Nvidia GPUs. ... Models have a maximum of 50 turns. We use the same prompt for all models, which is the default prompt from [16].” · Show in paper
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents· Method
PaperCodeSource excerpt
“Specifically, RQ3.1 analyzes the differences in agent steps, tool calls, and token usage across different configurations of NanoHarness, while RQ3.2 measures the distribution patterns of probing behavior and uses case studies to illustrate shifts in failure modes.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“As shown in Figure 10(c), synthetic distractors lead to slightly lower performance than the other types. This suggests that in-distribution distractors, which are structurally similar to relevant content, make the task harder compared to other distractors.” · Show in paper
Verified
Table 4 and Figure 10(c) directly show lower accuracy for synthetic distractors than for natural distractors or random noise.
reviewbench
PaperCodeDataSource excerpt
“R3 achieves the highest consequential rate across all three venues (87.9–93.2%), compared to 65.5–77.5% for Gemini 3 Pro (all padj < .001, |d| ≥ 0.79), 73.7–81.4% for GPT-5.2 (all padj < .001, |d| ≥ 0.50), and 60.0–67.7% for humans (all padj < .001, |d| ≥ 1.00) (Figure 5A).” · Show in paper
Verified
Figure 5A and its caption support the reported consequential rates, significance levels, and effect sizes.
BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence
PaperSource excerpt
“Strategy-induced stylistic drift is visible even when judge metrics remain stable. For ChatGPT, CoT does not materially change audit/accuracy but substantially reduces ROUGE-L (0.247→0.192), indicating more paraphrastic or verbose generations. For Qwen, CoT/reflection improve judge-based correctness but decrease ROUGE-L/BLEU (e.g., ROUGE-L drops by 0.042 under CoT), suggesting that reasoning prompts yield answers that are judged more correct yet less surface-aligned with the GT wording.” · Show in paper
Verified
Table 2 confirms the divergence between ROUGE-L/BLEU and judge-based metrics for ChatGPT and Qwen.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“Across five backbones and four benchmarks at a 10% GPU-resident KV budget, SEKV outperforms every method that fixes resolution at prefill, improving over the strongest semantic baseline by 2.9 points (5.5% relative).” · Show in paper
Verified
The averages in Table 1 and per-backbone breakdown in Table 7 directly support the 2.9-point (5.5% relative) improvement over SentenceKV and the superiority over all tested fixed-resolution baselines.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
PaperCodeSource excerpt
“Among the 124 runs in which an agent terminates on its own before the budget expires, a different limitation becomes clear: weak self-verification. We define a false finish as an early exit at relatively high reward, where the agent stops despite having not yet satisfied the hidden verifier. We identify 14 such runs with R >= 0.75.” · Show in paper
Verified
The counts, examples, and per-model early-exit mean rewards in Figure 6 and Section 3.5 support the claim.
fadebench
CodeVerified
The counts (1/10, 2/10, 9/10) and position audit match §5.4, and the authors appropriately note that the other three base models hold cold as well.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
PaperSource excerpt
“In Table 2 we see the frontier models solve about 99% of tasks, and each of the 642 tasks is solved by at least one of them, which independently confirms that every task is solvable.” · Show in paper
Verified
Both reference-patch checks and model evaluation results confirm that all 642 tasks are solvable.
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows· Method
PaperCodeDataSource excerpt
“The suite contains 274 general and 26 multimodal tasks across six categories and 17 subcategories (Figure 1). ... Across the 300 tasks, 1,408 scored modules assess delivery requirements, with three to seven modules per task.” · Show in paper