Search publications
Gemma 4 Technical Report· Method
PaperSource excerpt
“Our local to global attention ratio patterns follow Gemma Team (2025a), that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e. , values = keys. We encode position with p-RoPE with p = 0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model.” · Show in paper
QLoRA: Efficient Finetuning of Quantized LLMs
PaperCodeSource excerpt
“When using the standard practice of applying LoRA to query and value attention projection matrices [28], we are not able to replicate full finetuning performance for large base models. As shown in Figure 2 for LLaMA 7B finetuning on Alpaca, we find that the most critical LoRA hyperparameter is how many LoRA adapters are used in total and that LoRA on all linear transformer block layers are required to match full finetuning performance.” · Show in paper
Verified
Figure 2 clearly supports that QLoRA-All matches the 16-bit baseline while QLoRA-Attention falls short.
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“Since a single physical block can be mapped to multiple logical blocks, we introduce a reference count for each physical block. ... vLLM implements a copy-on-write mechanism at the block granularity for the physical blocks that need modification by multiple sequences, similar to the copy-on-write technique in OS virtual memory” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“The batched implementation adds 1.6–6.7% to the prefill time of the prefill anchor across the four pairs, whereas the per-layer implementation adds 5–51% (Figure 4b). ... The request takes 2.96 s, close to the 2.80 s of Llama-3.2-Instruct-1B alone and 3.0× faster than the 8.83 s of Llama-3.1-Instruct-8B alone (Figure 1).” · Show in paper
Verified
Measured median latencies on one A40 GPU in Figure 1, Figure 4, and Section 4.3 directly support the reported percentages and speedups.
End-to-End Context Compression at Scale· Method
PaperCodeSource excerpt
“After the first 768 generated tokens, we encode the oldest 512 generated tokens into latent memory tokens at a 16x compression ratio, resulting in 32 latent tokens, and keep the most recent 256 tokens uncompressed. Generation then continues from the compacted context and latest raw tokens, and the procedure repeats after every further 512 generated tokens.” · Show in paper
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers· Method
PaperCodeSource excerpt
“Our entire GPTQ calibration data consists of 128 random 2048 token segments from the C4 dataset (Raffel et al., 2020) ... Specifically, we always load one Transformer block, consisting of 6 layers, at a time into GPU memory and then accumulate the layer-Hessians and perform quantization. Finally, the current block inputs are sent through the fully quantized block again to produce the new inputs for the quantization of the next block.” · Show in paper
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
PaperCodeSource excerpt
“On 40 CPU threads with the checkpoint in the page cache, it takes 5 s on the 1B models, 12 to 13 s at 3 to 4B, 25 s at 7B and 37 s at 14B; reading the checkpoint from disk adds up to 21 s (released as profile_timing.json).” · Show in paper
Verified
Directly reported measurement from the authors' profiling pass.
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
PaperSource excerpt
“Experiments on NVIDIA H100 GPUs with Qwen-7B demonstrate up to 1.56× end-to-end throughput improvement over BF16, revealing a trade-off between throughput and model quality.” · Show in paper
Verified
Table 2 directly supports the 1.56× speedup at G = 256 under TP4 and demonstrates the throughput-quality trade-off across group sizes.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving· Method
PaperCodeSource excerpt
“To address this limitation, FlashInfer offers a head-group fusion strategy: different KV heads are mapped to individual threadblocks, while query heads are fused with the query length dimension.” · Show in paper
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs· Method
PaperCodeSource excerpt
“The rollout server loads the policy in the target low-bit format and uses an inference engine such as vLLM to generate responses with native low-precision kernels. Verifiers score these responses and send the resulting trajectories and rewards to the training server. The training server maintains high-precision latent weights and optimizer states in PyTorch.” · Show in paper
Programming a Coding Agent's Weights by Hand· Method
PaperSource excerpt
“Figure 1 shows the loop. Algorithm 1 (Appendix A) lists its six steps: diagnose, locate, write, predict, verify by replay and by agent evaluation, and score. The author is an LLM with shell access to the model weights, stored activations and logged episodes. Steps 1–4 run on a CPU; only verification uses a GPU.” · Show in paper
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“To verify that half-precision accumulation does not degrade quality in practice, we compare the LongBench scores obtained with FP16 and FP32 moment statistics on LLaMA-3.1-8B at L=128. The average score difference is less than 0.05 across all task categories, confirming that the numerical error introduced by half-precision accumulation is negligible relative to the approximation error O(\sigma^2).” · Show in paper
Verified
The reported experiment in Appendix C.2 supports the claim, though no per-category table is shown.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“The routing summary projects token keys into a -dimensional space with a learned per-head matrix and averages them with surprisal weights : .” · Show in paper
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PaperSource excerpt
“Overall, PagedWeight matches the throughput of the uniform APL-style baseline with similar or lower memory consumption. The largest throughput drop is 3.3% at batch size 1 and 4.1% at batch size 4.” · Show in paper
Verified
Table 3 confirms the throughput drops: at B=1, max drop is (134.5 - 130.1)/134.5 = 3.27% (3.3%); at B=4, max drop is (401.9 - 385.4)/401.9 = 4.11% (4.1%).
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“SEKV gains more from extra budget than ShadowKV, so its gap to ShadowKV narrows from 3.4 points at 5% to 1.0 at 20% on RULER and from 2.8 to 0.3 on NIAH, and on LLAMA-3.1-8B SEKV matches ShadowKV on NIAH at 20%.” · Show in paper
Verified
Table 4 directly supports all stated numbers.
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“1. Inference scaling is substantial but highly benchmark-dependent. Some benchmarks continue to improve well beyond typical published token budgets – including FrontierMath, TerminalBench, and Humanity’s Last Exam (HLE) – while others show weaker marginal gains under our protocol.” · Show in paper
Verified
Table 3 and Figure 1 directly substantiate both the substantial gains on FrontierMath and HLE and the weak marginal gains on SWE-Bench Pro, TerminalBench, and HealthBench under the tested protocol.
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“The longer agentic profiles exceed the single-H100 HBM budget: coding agent exceeds the wall by C = 40 and reaches 155 GB at C = 80, while terminal reaches 126 GB (Appendix Figure 4b clips both at the 80 GB wall).” · Show in paper
Verified
The analytical CF model in Figure 4b and Table 14b directly supports the capacity saturation thresholds.
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“As shown in Fig. 16 (a), vLLM achieves 1.67× higher throughput than Orca (Oracle) when the one-shot prefix is shared. Furthermore, when more examples are shared (Fig. 16 (b)), vLLM achieves 3.58× higher throughput than Orca (Oracle).” · Show in paper
Verified
Figure 16 directly supports the 1.67x and 3.58x throughput improvements over Orca (Oracle).
Gemma 4 Technical Report
PaperSource excerpt
“Combined with KV cache sharing (Shazeer, 2019) and the reuse of keys as values in global layers (Kayyam et al., 2026), these optimizations reduce the global KV cache footprint by up to 37.5%.” · Show in paper
Verified
The reduction follows directly from the specified architectural parameterization ( -RoPE and key-value reuse) and is consistent with the KV cache footprints reported in Table 3.
QLoRA: Efficient Finetuning of Quantized LLMs
PaperCodeSource excerpt
“(2) Double Quantization, a method that quantizes the quantization constants, saving an average of about 0.37 bits per parameter (approximately 3 GB for a 65B model).” · Show in paper
Verified
The mathematical derivation on page 5 is exact, and Figure 3 confirms that Double Quantization preserves zero-shot accuracy.