Search publications
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving· Method
PaperCodeSource excerpt
“At deployment time, a one-time profiling script collects data for the analytical models by running production traces at various request rates against the target model hardware combination. The step latency and batch contents are recorded at each step and used to train the analytical models. Total profiling time is ~30 minutes per model/hardware combination.” · Show in paper
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues.” · Show in paper
Verified
The experimental results in Tables 2 and 5 support the overarching finding that all evaluated models resolve only a tiny fraction of SWE-bench issues (under 2% for Claude 2, and 3.79% for Claude 3 Opus in the updated Table 5).
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Per-group rank allocation beats uniform rank at all 15 grid cells (ratios 2, 3, 4, 6, 8 at three context lengths), with a peak advantage of 0.97 PPL at n=512, 4×.” · Show in paper
Verified
Tables 12 and 13 confirm all reported perplexity differences.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
PaperSource excerpt
“Experimental results validate our theoretical findings, showing that for KV cache quantization, we achieve absolute quality neutrality with 3.5 bits per channel and marginal quality degradation with 2.5 bits per channel.” · Show in paper
Verified
The experimental results in Figure 4 and Table 1 directly support the claimed quality neutrality at 3.5 bits and marginal degradation at 2.5 bits.
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
PaperCodeDataSource excerpt
“Harness choice substantially affects performance within individual models. General-task score gaps across harnesses range from 5.0 to 9.5 points within nine models, and reach 39.2 for Qwen (Table 2). ... Multimodal gaps are larger for several models: Kimi ranges from 28.1 with Codex to 72.7 with OpenClaw, with preferred harnesses differing across backends (Table 3).” · Show in paper
Verified
The reported ranges match the values in Tables 2 and 3.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading· Method
PaperCodeSource excerpt
“We evaluate models using the Harbor framework with the Terminus-2 agent harness [10], which interacts with each task environment through a single long-horizon terminal session. Additionally, for GPT-5.3, we instead use Codex [28] as the agent harness.” · Show in paper
BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence· Method
PaperSource excerpt
“We evaluate representative MLLMs, including closed-source systems (Gemini-3 [5], ChatGPT-5[15]) and open-weight models (Qwen3 [1], Gemma-3 [4]), under a unified evaluation pipeline. All models are tested with standardized input formatting and identical decoding settings (temperature = 0.2, max tokens = 2048) to ensure fair comparison. For retrieval-augmented experiments, we employ ColPali as a multimodal retriever at the page level, retrieving top-k = 3 relevant pages per query. Retrieved content is concatenated with the prompt before generation. Then for result evaluation, we adopt Qwen-Plus (temperature = 0.2) as an LLM-as-a-judge to score final answer correctness and evidence alignment on a 0–5 scale. We further analyze reasoning behaviors using a structured error taxonomy. In addition, we report ROUGE [11] and BLEU [17] as complementary automatic metrics.” · Show in paper
fadebench
CodeVerified
The counts (1/10, 2/10, 9/10) and position audit match §5.4, and the authors appropriately note that the other three base models hold cold as well.
How Inference Compute Shapes Frontier LLM Evaluation· Method
PaperSource excerpt
“For total budget B, we estimate pass@k by splitting B across k independent trajectories of B/k tokens each and counting a task as solved if any trajectory’s first correct submission falls within budget. pass@k is computed task-wise using the unbiased estimator of Chen et al. (2021).” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression· Method
PaperCodeSource excerpt
“In contrast, we propose hybrid sparse attention (HSA), which takes advantage of two-dimensional reduction, in both sequence and head dimensions together to achieve better estimation accuracy on KV token indices with top-k attention scores.” · Show in paper
Resource-Efficient Speculative Decoding for Long-Context LLM Serving
PaperSource excerpt
“In Fig. 7, streaming access for the full query group yields the largest throughput gain. With 16K inputs, K2 improves throughput over K1 by an average of 83.2% on Qwen3 and 117.1% on InternLM2.5.” · Show in paper
Verified
The ablation results in Figure 7 and Section 5.3.1 directly substantiate the individual and cumulative gains of streaming verification.
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“We further analyze Yi-34B, which claims context length of 200K and achieves reasonably good performance on RULER among open-source models. Our results demonstrate large degradation in Yi's performance as we increase input length and task complexity.” · Show in paper
Verified
Figures 2 and 3 and the quantitative error analysis in Section 5 substantiate each of the reported failure modes.
Gemma 4 Technical Report
PaperSource excerpt
“Gemma 4 31B is closest in size and significantly better across the board, while E2B roughly matches Gemma 3 27B performance with 10x less parameters.” · Show in paper
Verified
Table 5 supports both parts of the claim, noting that Gemma 4 models were evaluated in thinking mode whereas Gemma 3 27B was evaluated in non-thinking mode.
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs· Method
PaperCodeSource excerpt
“the original high-precision pretrained checkpoint serves as a frozen teacher to supervise a low-precision student initialized from the same weights. Given an input sequence of length L, the framework minimizes the token-averaged forward Kullback–Leibler (KL) divergence between the teacher distribution pT,t and the student distribution pS,t over their shared vocabulary V” · Show in paper
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs· Method
PaperSource excerpt
“Trace replay profiles preserve the exact recorded requests from the source traces. Distributional synthetic profiles sample multi-turn requests from per-turn ISL, OSL, and turn-count distributions fitted to those traces, producing representative workloads that can be reissued across models, hardware, backends, and concurrency settings, without the runtime cost of full replay.” · Show in paper
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving· Method
PaperCodeSource excerpt
“Query and output matrices are efficiently stored as ragged tensors (also known as jagged arrays) (Tensorflow Developers, 2018) without padding, which facilitates the compact packing of queries and outputs from diverse requests into a single tensor.” · Show in paper
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K.” · Show in paper
Verified
The results in Table 3 and Tables 10-11 directly support the claim; 10 of 17 models (roughly half) pass the 85.6% threshold at 32K, while nearly all achieve near-perfect scores on vanilla NIAH.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction· Method
PaperCodeSource excerpt
“Each gate first computes a head-specific embedding, but the final scalar score is produced by a shared projection: gℓ,h(x) = σ (w⊤g Projℓ,h(x) + bg)” · Show in paper