Search publications
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“Higher mean performance is associated with smaller variation (Figure 3(a)): Spearman’s ρ = −0.76, or −0.67 excluding Qwen. Full delivery remains sensitive, however: GPT Sol has the highest mean and an SD of only 1.93, yet its full-credit rate varies by 9.12 percentage points. Both GLM variants have full-credit gaps of 13.50 points.” · Show in paper
Verified
Table 9 and Figure 3(a) directly support both the negative correlation between mean score and SD and the persistent sensitivity of R1.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“LCLMs achieve the fastest compression time at every context length and substantially lower peak memory at long contexts; all baselines run out of memory at 512K tokens, except Attention Matching, which fails at 512K due to numerical instability in its linear solver and runs out of memory at 1M.” · Show in paper
Verified
Figure 4 and Section 6.2 clearly document the scaling measurements and memory limits on the H200 GPU.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces· Method
PaperSource excerpt
“A Terminal-Bench task consists of an instruction, a Docker image, a set of tests, an example solution, and a time limit (Figure 2).” · Show in paper
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime—a ∼10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes—while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction).” · Show in paper
Verified
The paired experiments, no-cache and session-cap controls, and reproductions across models and engines directly support the claim of metastability.
How Inference Compute Shapes Frontier LLM Evaluation· Method
PaperSource excerpt
“Our main experiment follows a fully crossed design with six frontier models (Table 1) evaluated on five non-cyber benchmarks under two feedback conditions, using one shared ReAct-style scaffold (Yao et al., 2023) implemented in Inspect AI (UK AI Security Institute, 2024a). For each task in each (benchmark, model, condition) cell, we run 5 independent trajectories.” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“As shown in Figure 9, the best static split factor varies with different sequence lengths and token budgets, while adaptive compression decomposition provides comparable accuracy to the best static split factor in most cases.” · Show in paper
Verified
Figure 9 shows that adaptive RocketKV closely matches or exceeds the best static split factor across the tested benchmarks and token budgets.
How Inference Compute Shapes Frontier LLM Evaluation· Method
PaperSource excerpt
“For each benchmark, we fit a linear probability model regressing on model generation, leave-one-model-out task difficulty, and their interaction, by ordinary least squares (OLS) with cluster-robust standard errors at the task level.” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme.” · Show in paper
Verified
Figure 6 and Table 4 on SCBench directly demonstrate that RocketKV-MT outperforms all non-oracle baselines and closely matches Exact-TopK across all tested token budgets.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
PaperSource excerpt
“with scaffold, language and run length held constant, testing after the last edit roughly doubles the odds of solving for all three models. The median failed run writes 1.6–2.0 times the completion tokens of the median solved run in the same model–scaffold pair (Figure 4c): a failing run is one that kept going, not one that gave up.” · Show in paper
Verified
Figure 4c and Appendix B support the completion-token ratios and classifier-based test-after-edit finding.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“Our results show that the effective context size is approximately 16k tokens for the best prompt-only and 32k tokens for the best RAG method. This suggests that the model is able to effectively use only up to 25% to 30% of its advertised context length.” · Show in paper
Verified
Table 1 directly supports the 16K and 32K ECW values under the authors' 30% degradation threshold.
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“FrontierMath’s commonly used budget of 1M total tokens (Burnham, 2025) is itself an order-of-magnitude increase over the earlier 100k-token scaffold (Glazer et al., 2024), and extending from 1M to 10M adds a further +11.7 ± 11.0 percentage points on average. HLE also continues to improve well beyond its typical 64k-output-token budget, gaining +11.9 ± 10.4 points up to our 5M cap.” · Show in paper
Verified
The reported gains match Table 3 (rounded to one decimal place in the text) and are properly qualified as properties of the tested serial scaling protocol.
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
PaperCodeDataSource excerpt
“Qwen2.5-Coder-32B-Instruct underperforms expectations, especially considering its strong code generation capabilities. Analysis of its trajectories reveals problems with instruction following; the model frequently hallucinated environment responses or enters loops of formatting errors, ultimately failing without producing a meaningful solution attempt.” · Show in paper
Verified
Both the quantitative underperformance in Table 1 and the qualitative failure mode in Appendix J.2 are directly documented.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
PaperSource excerpt
“Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch.” · Show in paper
Verified
The task counts, defect taxonomy, and repair pipeline are consistently documented in Figure 2, Table 1, and Sections 3.1–3.5.
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
PaperCodeSource excerpt
“Under that control the ranking has no measurable advantage (Appendix N lists every setting). Across five models and three budgets, top-K beats the median matched draw in 11 of 15 settings, and 50 of the 150 draws beat top-K where chance predicts 75. A sign test does not separate that from chance (p=.12), and the median gap is 0.01 perplexity.” · Show in paper
Verified
The parameter-matched control in Table 19 and downstream comparisons in Section 7.2 directly support this negative result.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces· Method
PaperSource excerpt
“Between the three reviews, the average task received approximately three hours of combined reviewer attention, implying multiple hundreds of person-hours went into reviewing alone, excluding the time spent creating the tasks.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Table 1 reports perplexity at n=1024 (T=512) on all five models. Every free-zone cell stays below the 0.2% threshold, with the largest drift at +0.17% (Qwen at 3×). The free zone holds at context lengths 512 and 2048 as well (Appendix B).” · Show in paper
Verified
All reported 2x and 3x perplexity values in Table 1 and Tables 4-6 are strictly below +0.2% relative drift.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
PaperSource excerpt
“We show that frontier models and agents score less than 65% on the benchmark and conduct an error analysis to identify areas for model and agent improvement.” · Show in paper
Verified
The experimental results in Figure 1, Section 4, and Table 2 directly support that all evaluated frontier models and agents score below 65%.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“Applying the audit workflow to Terminal-Bench 2.0 surfaced issues in 28 tasks, roughly a third of the benchmark. ... Fig. 3 shows the breakdown of the task defects in the 28 tasks that needed repair. The most common failure mode was instruction–verification mismatch (present in 57% of the tasks). ... The second most common defect was task drift (43% of tasks). ... The third category, environment issues (29% of tasks), comes from resource budgets.” · Show in paper
Verified
The counts in Table 2 (16/28 = 57.1%, 12/28 = 42.9%, 8/28 = 28.6%) match the reported percentages, and Table 3 documents the exact repairs per task.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Agentic LLM workloads occupy a hardware regime that no prior inference benchmark captures. Across 22 workload profiles, 9 models, and 14 GPU configurations (3,197 sweep rows, 148,077 NCU records), AGENTPERFBENCH identifies a dual pressure that separates agentic serving from chat.” · Show in paper
Verified
The benchmark comparison, saturation sweeps, and roofline/capacity analyses directly support the claim that agentic workloads exhibit this dual-pressure regime uncaptured by prior benchmarks.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“To evaluate whether the workflow transfers beyond Terminal-Bench, we applied it to 34 randomly selected tasks from an 80-task snapshot during the development of Harbor-Index v0.1. ... The workflow flagged 14 tasks, of which manual review confirmed 12 as defective (86% precision).” · Show in paper
Verified
12/14 = 85.7% (~86%) precision is directly supported by the manual review of all 34 tasks.