Search publications
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks· Method
PaperCodeDataSource excerpt
“Specifically, given a task instance (passage, query) and a corpus of background texts, we retrieve the top-k most semantically similar distractor passages based on cosine similarity. The value of k is adjusted dynamically to control the overall length of the prompt, allowing our setup to scale the long contexts to any arbitrary length as needed.” · Show in paper
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“Training at scale confirms the choices from the small-scale sweep: both the continual pre-training losses (Figures 9 to 11) and the downstream ablations (Appendix F.6) again favor an encoder window W = 1024 and causal masking. At scale, the attention-based adapter is competitive with the MLP adapter; we keep the MLP adapter, which adds less computation. The choice of pooling operator depends on the compression ratio: mean pooling has a small edge over concatenation at 16x compression, while concatenation slightly outperforms mean pooling at 4x.” · Show in paper
Verified
Both the from-scratch pre-training sweeps and the full-scale continual pre-training and downstream ablations support these architectural conclusions.
Measuring AI Ability to Complete Long Software Tasks
PaperSource excerpt
“In addition, we linearly regress log(time horizon) against release date, finding that time horizon has doubled every 207 days with a 95% bootstrapped confidence interval 166-240 days (roughly ±19%).” · Show in paper
Verified
The data, hierarchical bootstrap confidence intervals, and comparison against linear and hyperbolic fits support the exponential doubling trend on this task suite.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“The unquantized Qwen3-1.7B control shows the same qualitative monotonic trade-off: paraphrase EM rises from 76% at r = 8 to 79% at r = 16 and 86% at r = 32, while average OOD performance falls from 57.0% to 52.0% and 40.2%, respectively. This suggests that quantization is not required for the qualitative rank trend, although this reduced control changes model scale.” · Show in paper
Verified
The reported numbers on Qwen3-1.7B demonstrate the same monotonic trade-off without quantization, and the authors appropriately qualify the difference in model scale.
BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence· Method
PaperSource excerpt
“We evaluate representative MLLMs, including closed-source systems (Gemini-3 [5], ChatGPT-5[15]) and open-weight models (Qwen3 [1], Gemma-3 [4]), under a unified evaluation pipeline. All models are tested with standardized input formatting and identical decoding settings (temperature = 0.2, max tokens = 2048) to ensure fair comparison. For retrieval-augmented experiments, we employ ColPali as a multimodal retriever at the page level, retrieving top-k = 3 relevant pages per query. Retrieved content is concatenated with the prompt before generation. Then for result evaluation, we adopt Qwen-Plus (temperature = 0.2) as an LLM-as-a-judge to score final answer correctness and evidence alignment on a 0–5 scale. We further analyze reasoning behaviors using a structured error taxonomy. In addition, we report ROUGE [11] and BLEU [17] as complementary automatic metrics.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“Overall, these results indicate that KV cache compression techniques introduce only limited additional latency for the first token, with the exception of TurboQuant, whose more complex quantization pipeline leads to noticeable overhead.” · Show in paper
Verified
Table 3 and Figure 2 support the claim for the evaluated datasets, though GovReport results for TurboQuant are omitted due to 3-day single-run execution times.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“Figure 2 shows that configurations with stronger OOD degradation also exhibit larger drift from the pretrained model. Higher-rank QLoRA checkpoints have larger symmetric KL divergence and larger effective dense update magnitudes. The strongest forgetting regime, QLoRA r=64, also has the largest SVD intruder excess, indicating a larger change in the leading spectral structure of adapted weight matrices.” · Show in paper
Verified
Figure 2 clearly shows monotonic increases in symmetric KL, dense RMS drift, and SVD intruder excess with QLoRA rank, exceeding the FFT baseline.
BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence
PaperSource excerpt
“Using audit_score as the primary metric, ChatGPT performs best across strategies (4.379–4.404), followed by Gemma (4.131–4.175), Gemini (3.728–3.989), and Qwen (3.489–3.605). Strategy effects are model-dependent: Gemini performs best with direct prompting and degrades under CoT/reflection (audit drops by 0.198 and 0.261, respectively), while Qwen improves with CoT/reflection (audit +0.116/+0.105).” · Show in paper
Verified
All reported audit_score ranges and strategy-dependent deltas match the values in Table 2.
reviewbench· Method
PaperCodeDataSource excerpt
“To assess whether the evaluating LLM introduces self-preference bias, we randomly sampled 50 papers and assessed each source’s comments using GPT-5.2 with the same prompt, claims, and post-processing pipeline used by the primary Gemini evaluator.” · Show in paper
fadebench
PaperCodeSource excerpt
“Read as a detector of drift, it is a poor one: 230 of the 273 drift steps are not marked down. mistral-nemo-12b marks about half its drift steps down, but also about a quarter of its holds and third courses. On phi-4-mini and qwen3-8b the third course is marked down at least as often as drift. So on none of the four base models is the retrospective a reliable detector of drift.” · Show in paper
Verified
All counts and percentages match §5.12 and Figure 4.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving· Method
PaperCodeSource excerpt
“For task quality evaluation, we use six representative datasets: HotpotQA and 2WikiMQA for multi-document QA, Qasper and MultiFieldQA_en for single-document QA, TriviaQA for few-shot learning, and MultiNews for summarization.” · Show in paper
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“Across 63 comparable chat groups, median request throughput grows +11%, with 33 groups still improving by more than 10%. In contrast, the 66 comparable agentic groups are mostly already at or past saturation by C=200: 41 remain within ±10% at C=320, while 15 lose more than 10% throughput.” · Show in paper
Verified
The aggregate counts across 63 chat and 66 agentic groups, along with the representative configurations in Table 5, support the claimed scaling divergence.
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
PaperSource excerpt
“As shown in Table 3, Verified+ achieves the best F1 score, while Verified-E achieves the highest precision. ... Table 5 demonstrates that ensembling by averaging scores improves robustness and yields the best overall F1, whereas strict consensus is beneficial when precision is prioritized.” · Show in paper
Verified
Tables 3 and 5 directly support the reported precision and F1 trade-offs across prompt and ensemble configurations.
End-to-End Context Compression at Scale· Method
PaperCodeSource excerpt
“An LCLM consists of an encoder with a pooling method that maps token chunks to soft tokens, an adapter to project to the decoder's embedding dimension, and a decoder that consumes the soft tokens as latent context.” · Show in paper
BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence
PaperSource excerpt
“Table 5 shows a clear degradation as the required evidence moves deeper into the document. For all models, performance is highest when evidence appears early (pages 1–2) and drops for later bins (pages 11–20 and 21+).” · Show in paper
Verified
Table 5 supports the degradation trend across page depth for ChatGPT and Qwen (with the noted small-N caveat for Qwen at 21+); Gemini's numbers are cited in the text though omitted from Table 5.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“while transferring 3× less data per decoding step than ShadowKV and using the least total memory among the methods we profile. At 128K context, it halves peak GPU memory and cuts total memory by one third relative to full KV caching.” · Show in paper
Verified
Tables 2 and 20 directly confirm the 3.1x transfer reduction vs. ShadowKV, lowest total memory (46.01 GB), ~50% peak GPU memory reduction, and ~34% total memory reduction vs. Full KV.