Search publications
RULER: What's the Real Context Size of Your Long-Context Language Models?· Method
PaperCodeSource excerpt
“We evaluate eight open-sourced models at various context sizes across 18 task configurations. Each task can then be represented with a vector of model performance at various context sizes. The 18 task vectors are then clustered via agglomorative clustering algorithm, using correlation coefficient as the distance metric.” · Show in paper
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning· Method
PaperCodeSource excerpt
“Given a protection budget ρ ∈ [0, 1] (sensitive-frac), the K=⌈ρL⌉ layers with largest εℓ are loaded in fp16; all others in NF4 with double quantization; lm_head stays fp16.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“FlashJoLT replaces this SVD with a randomized approximation [11] that computes only the leading qcap token-mode singular vectors. The feature-mode SVD is kept exact because dh is small (128), and the rest of the pipeline, including the Lagrangian allocator, is unchanged.” · Show in paper
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“Our additional ablations demonstrate that training on longer sequences does not always lead to better performance on RULER” · Show in paper
Verified
Both the LWM ablation in Figure 4 and the main results in Table 3 support that longer training context does not guarantee superior RULER performance.
RULER: What's the Real Context Size of Your Long-Context Language Models?
PaperCodeSource excerpt
“Figure 4 (middle-right) shows that the 34B model is significantly better than the 6B model on RULER for both performance at length of 4K and the relative degradation, suggesting the benefit of scaling model sizes for better long-context modeling.” · Show in paper
Verified
Figure 4 (middle-right) clearly shows Yi-34B-200K outperforming Yi-9B-200K and Yi-6B-200K across all tested lengths.
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
PaperCodeSource excerpt
“Evaluation results demonstrate that our method substantially narrows the accuracy gap, achieving on average a 70.63% reduction in accuracy loss compared to MXFP4 and a 37.30% reduction relative to the latest NVFP4 on LLM benchmarks. Furthermore, our design delivers up to 1.91× speedup and 1.75× energy savings over state-of-the-art accelerators.” · Show in paper
Verified
The reported 70.63% and 37.30% accuracy loss reductions match the Table 2 averages, and the 1.91x speedup and 1.75x energy savings are supported by the normalized latency and energy comparisons in Figure 13.
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
PaperSource excerpt
“The results demonstrate that FP4Quant consistently outperforms Tahquant in preserving model quality under FP4 communication compression. Under TP, Tahquant suffers from severe accuracy degradation (loss 3.4848, PPL = 32.6163), whereas FP4Quant with G = 64 achieves substantially lower loss (2.2306, PPL = 9.3050) and maintains a stable convergence trajectory close to the full-precision baseline, as illustrated in Fig. 5. Under PP, FP4Quant further demonstrates near-lossless performance (loss 2.1525, PPL = 8.6105), matching BF16 accuracy and outperforming Tahquant.” · Show in paper
Verified
The numerical metrics in Table 2 and convergence curves in Figure 5 support the claim for G = 64 (and G <= 256, as qualified on page 9).
Measuring AI Ability to Complete Long Software Tasks
PaperSource excerpt
“GPT-2 has a 50% time horizon of only 2 seconds, while o3 has a 110-minute time horizon and succeeds at several tasks over 4 hours. Also, o3 lies above the long-run trend (p = 0.006); this is robust to methodological ablations like using continuous scoring (Appendix H), which may imply the trend in 2024 and early 2025 is faster.” · Show in paper
Verified
The logistic regression estimates, p-value, and 2024-2025 comparison support the claim as qualified with 'may imply'.
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“Relative to the unedited model, the residual-stream changes of adapter and hook have cosine 0.972–0.992 and 0.984–0.996 across eight layers from 18 to 41. Their norm ratios are within 0.4% of 1. Over completion tokens, KL(hook k adapter) is 1.1 × 10−4 and 1.2 × 10−4. This is 99 and 110 times smaller than the edit’s own effect, KL(unedited k adapter) = 0.011 and 0.013.” · Show in paper
Verified
The teacher-forced forward-pass measurements in Table 12 directly support the claimed cosine similarities, norm ratios, and KL ratios.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“The partial Tucker approximation X˜(rT , rd) computes the mode-k singular vectors of the token and feature unfoldings via the sequentially truncated higher-order SVD (ST-HOSVD) [22] and truncates them to ranks rT and rd, while keeping identity factors on the layer and head modes.” · Show in paper
Resource-Efficient Speculative Decoding for Long-Context LLM Serving· Method
PaperSource excerpt
“The Target KV Cache is partitioned into the following three regions accordingly, Sealed History: [0,H) (CPU) Active Tail: [H,Cr) (GPU) Speculative Frontier:[Cr ,Lr)(GPU) 0 ≤ H ≤ Cr ≤ Lr.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“The base LLM stays frozen. We train only the projections , the thresholds , and the rank predictor , under 0.1% of the backbone’s parameters, once per backbone on generic text (Section 4.1). The objective combines four terms: .” · Show in paper
Gemma 4 Technical Report· Method
PaperSource excerpt
“In our MTP procedure, the model's last layer activations from the previous step and token embeddings are fed into the MTP head. The MTP head generates future tokens sequentially using a separate embedder and a 4-layer Transformer block that cross-attends to the KVs of the main model (Figure 1), thus eliminating the need for MTP prefill and supporting any draft length.” · Show in paper
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“Figure 2 shows that configurations with stronger OOD degradation also exhibit larger drift from the pretrained model. Higher-rank QLoRA checkpoints have larger symmetric KL divergence and larger effective dense update magnitudes. The strongest forgetting regime, QLoRA r=64, also has the largest SVD intruder excess, indicating a larger change in the leading spectral structure of adapted weight matrices.” · Show in paper
Verified
Figure 2 clearly shows monotonic increases in symmetric KL, dense RMS drift, and SVD intruder excess with QLoRA rank, exceeding the FFT baseline.
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
PaperCodeSource excerpt
“M2XFP achieves the lowest perplexity on all models except OPT-6.7B, where BlockDialect is better by only 0.03.” · Show in paper
Verified
Tables 3 and 4 directly substantiate the perplexity rankings and the reasoning accuracy recovery.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition–retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime.” · Show in paper
Verified
The five-seed results in Figure 1 and Table 5 directly support the full claim, including the relative positions of FFT and QLoRA ranks 8, 16, 32, and 64.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs· Method
PaperSource excerpt
“We evaluate serving with a closed-loop concurrency sweep. At concurrency C, the benchmark harness admits at most C requests to the endpoint at a time. When one request completes, the next request or session turn is dispatched. For multi-turn profiles, each slot advances a session in turn order, preserving the session-context snowballing that makes agentic prompts grow across turns, increasing prefix-cache pressure.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“In summary, our results show that while LLMs are effective at retrieving isolated facts, their capabilities degrade substantially when required to reason over long contexts.” · Show in paper
Verified
Figures 1, 5, 6, and 8 clearly demonstrate the performance gap between retrieval tasks and reasoning tasks across all tested languages.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models· Method
PaperCodeSource excerpt
“We get a suitable α by running a quick grid search on a subset of the Pile (Gao et al., 2020) validation set. To get the statistics of activations, we calibrate the smoothing factors and the static quantization step sizes once with 512 random sentences from the pre-training dataset Pile, and apply the same smoothed and quantized model for all downstream tasks.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“KIVI4 provides the most stable quality across models” · Show in paper
Verified
Table 2 confirms that KIVI4 avoids the large accuracy drops seen in TurboQuant, SnapKV, and CaM on summarization while maintaining competitive or superior scores on QA and few-shot tasks.