Search publications
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage.” · Show in paper
Verified
The accuracy, TTFT, and peak memory benchmarks across RULER, LongBench, LongHealth5, and GSM8K support the claim that LCLMs improve the Pareto frontier over KV cache compression baselines.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“Even under optimal configurations, the accuracy of RAG-based methods at longer context windows remains approximately 20% − 40% below the best short-context baseline. This indicates that while RAG improves stability and extends effective context size, there remains considerable room for further advancement in handling extreme context lengths.” · Show in paper
Verified
Table 1 and Figure 9 support both the improved stability of RAG and its remaining 20%-40% gap relative to the 0k CoT baseline (0.598).
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents· Method
PaperCodeDataSource excerpt
“To initiate the dataset building process, we download approximately 450,000 pull requests linked to issues created before May 1, 2025. These originate from over 30,000 repositories that feature permissive licenses granting broad usage rights (see Appendix D for the list of included license types) and where Python constitutes over 75% of the codebase lines of code.” · Show in paper
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“Online latent compaction is nearly lossless, reaching 90.52% accuracy compared with 92.46% for full-context generation, while being highly efficient: it reduces the average KV length by 3.1x, the average per-problem peak by 3.6x, and the maximum peak by 7.9x: the longest full-context trajectories reach the 32,768-token limit, whereas no compacted trajectory exceeds 4,135 tokens.” · Show in paper
Verified
The full 5,000-problem MATH test set evaluation under matched prompts, context limits, and scoring directly supports the accuracy and KV-reduction claims.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
PaperCodeSource excerpt
“Across all models, 79% of unresolved runs (518/660) end because the 90-minute budget expires while the agent is still actively working. However, these timed-out runs are not close to completion, with mean reward ranging only from 0.10 to 0.35 across models.” · Show in paper
Verified
Figure 6 and Section 3.4 directly support that 79% of unresolved runs time out with mean rewards between 0.10 and 0.35.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“We show that fixed wall-clock and output-token budgets can materially change apparent agent performance: For example, extending the time limit closes a 20 pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.” · Show in paper
Verified
Figure 6 directly shows the 20.3 pp gap on GPT 5.4 closing completely (74.9% vs 75.3%) and the 29.9 pp gap on GPT 5.4 mini shrinking to 5.3 pp (68.2% vs 62.9%) at 32x timeout.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks· Method
PaperSource excerpt
“For each agent–model pair, we increase the wall-clock budget up to 32× the original task timeout and measure how many tasks are solved as the budget increases.” · Show in paper
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers.” · Show in paper
Verified
The SGLang paired runs and 32B trace-shape runs support the generalization, and the authors appropriately note that 32B provides weaker evidence of history dependence than 8B.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
PaperCodeSource excerpt
“Dense grading also reveals near-misses (0.75 <= R < 0.95) runs that are close to completion but fail some final checks. These near-misses occur nearly twice as often as passes (90 vs. 50), indicating that many runs get very close to fully solving the task, but fail at the last few steps or checks.” · Show in paper
Verified
The counts in Figure 4 (31 + 59 = 90 vs. 50) match the stated claim.
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“The five leading harnesses have similar pooled mean scores, but differ in complete delivery and expenditure (Table 4). OpenCode leads mean score; OpenClaw leads full-credit attainment at the lowest API cost.” · Show in paper
Verified
Table 4 directly supports the stated ordering of OpenCode and OpenClaw.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Chat stays below it, peaking at OIeff = 181 (Table 12), remaining memory-bandwidth-bound across the full concurrency sweep. The other agentic workloads cross the ridge: computer-use reaches OIeff = 886 (3.0x the ridge), terminal reaches 1,809 (6.1x), and coding agent reaches 2,409 (8.2x) at C = 80, all compute-bound. Peak separation between coding agent and chat is 13x.” · Show in paper
Verified
Table 12 and Figure 4a directly substantiate the exact OI values and ridge multiples on H100 for LLaMA-3.1-8B.
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“3. Gains from inference scaling do not arise from one universal intervention. Benchmark scores depend partly on protocol choices about if and how models can iterate on their solutions, and whether compute is allocated to a single deep trajectory (“serial” scaling) or spread across multiple shallower ones (“parallel” scaling). Repeated submissions materially improve performance on all benchmarks, and feedback on submission correctness matters most where it can guide continued search (HLE and SWE-Bench Pro). Parallel scaling is strongest on stateless benchmarks (HealthBench and HLE, which do not involve a persistent interactive environment) and weakest on stateful ones.” · Show in paper
Verified
Tables 5, 7, and 12 directly support all three components of this claim.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
PaperCodeSource excerpt
“OPUS 4.1 primarily fails on semantic understanding, with wrong solutions accounting for 35.9% of failures and syntax errors at 24.2% ... SONNET 4 has context overflow as its primary failure mode (35.6%) and substantial endless file reading behaviors (17.0%) ... QWEN3 32B, as an open-source model, exhibits the highest tool error rate (42.0%)” · Show in paper
Verified
Table 4 provides the exact counts and percentages for each failure bucket across the evaluated models, supporting the characterization of distinct failure modes.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
PaperSource excerpt
“Despite this wide variation in interaction patterns, we find essentially no correlation between the number of average turns per trial and model success rates (Figure 35). Similarly, while models vary significantly in their token generation patterns, our analysis shows that higher token count does not necessarily correlate with better performance (Figure 36).” · Show in paper
Verified
Both Appendix G.1 and G.2 show statistically non-significant correlations close to zero, supporting the claim despite minor numerical discrepancies between the text and figure legends.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Table 3 quantifies this mismatch under a controlled single-turn comparison: matched on model, hardware, backend, tensor-parallel degree, and C=80, coding-agent planning calls have a median 2.8x higher TTFT and 1.3x higher TPOT than chat across 40 released trace_replay groups.” · Show in paper
Verified
Table 3 directly reports the 2.8x median TTFT ratio and 1.3x median TPOT ratio across 40 matched pairs at C=80.
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
PaperSource excerpt
“Across all evaluated models, Code A tasks have substantially higher pass rates than B* tasks. For example, Gemini reaches 34.0% pass@3 on Code A but only 4.0% on Code B*, while GLM-4.7 reaches 34.0% versus 6.0%. This gap supports the intended use of the metadata: A-category tasks provide cleaner learning signals, while B-category tasks identify settings where tests, specifications, or environments introduce confounding factors.” · Show in paper
Verified
Table 7 shows a consistent, large gap in pass@1 and pass@3 between Code A and Code B* tasks across all five evaluated models.
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
PaperSource excerpt
“Table A1 shows that larger declared dependency sets are associated with lower first-attempt success in Python, Java, and C++. The strongest negative association appears in C++, while JavaScript shows the same direction but not a statistically meaningful effect in the current benchmark.” · Show in paper
Verified
Table A1 provides the exact Spearman correlation coefficients and p-values supporting the negative association in Python, Java, and C++.
Scaling Inference-Time Compute with Teams of Coding Agents
PaperSource excerpt
“Directed toward planning, sixteen Luna workers come within 1.4 points of an all-Sol team at 70% lower cost (Figure 8), making them roughly 4× more efficient.” · Show in paper
Verified
The data in Figure 8 and Section 7 directly support the 1.4-point gap and 70% cost reduction, though Figure 8 notes that fixed-size team runs are single runs compared to three runs for dynamic teams.
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“With DeepSeek Flash, Codex scores 68.34 at $0.02022 per task, versus Nanobot’s 60.92 at $0.03272. Nanobot’s higher cost coexists with extensive cache reuse (85.65% of input tokens). With GPT Sol, Nanobot spends 2.97 times as much as OpenClaw while achieving lower full-credit attainment (24.82% versus 31.02%).” · Show in paper
Verified
Table 17 and Table 7 confirm the exact cost ratios, scores, and full-credit rates.
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
PaperSource excerpt
“For GLM-5.3 and Kimi K3, accuracy barely moves: only one of their six scaffold pairs differs significantly, Kimi K3’s Pi against OpenCode by 1.7 points. Inkling is the exception, with Pi trailing the other two scaffolds by 5.8–7.5 points.” · Show in paper
Verified
Table 3, Table 4, and Figure 4a directly substantiate the accuracy spreads, p-values, and token-cost differences.