Search publications
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“The base LLM stays frozen. We train only the projections , the thresholds , and the rank predictor , under 0.1% of the backbone’s parameters, once per backbone on generic text (Section 4.1). The objective combines four terms: .” · Show in paper
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization· Method
PaperSource excerpt
“PagedWeight quantizes a network at the granularity of per expert linear-block (a weight in an expert). Each linear-block is identified by a triple index: i := (ℓ, e, u) for layer ℓ, expert e, and one of two linear-block types u ∈ {gate_up, down}. PagedWeight organizes linear-blocks and their quantization decision in a weight page table, where each linear-block i occupies one row.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“We will add the same vector to both tail tests; its contribution to the endpoint average will cancel. To control its size, at each gate g put mg = sup w⪰g |k([g, w])|.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“One unit of logarithmic address resolution costs one unit of entropy, whereas one unit of monochrome rank costs at least β units of probability. This strict surplus will bound total address occupancy and force an acyclic component at a suitable scale.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“We construct a replica path, starting at the same position as X, that will be on reference trajectories of unobserved labels.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“In each occupied bin 2a ≤ Ms < 2a+1, a ≥ 0, choose the finest partition that occurs in that bin. List the chosen partitions in order from fine to coarse as P0, . . . , Pℓ−1, with crossing sets Bi and sizes Mi.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“We bound the remaining uncertainty by successively refining those cells. The refinement cuts are random, independent of the samples, to prevent atoms close to a fixed boundary from being charged too often.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“Given a total byte budget B for the compressed prefill cache, JoLT minimizes the summed error subject to the summed cost: min {(rT ,rd,b)g,t} X g,t eg,t(rT , rd, b) s.t. X g,t sg,t(rT , rd, b) ≤ B.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For feasibility, form a directed network with a source, the row and column vertices, and a sink. Give the source-to-row arcs capacities ri, the row-to-column arcs capacities bij , and the column-to-sink arcs capacities cj . An integral flow of value N = P i ri = P j cj is exactly a feasible table” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For the second block, reveal only which slots contain each of two colors at every layer. That observation leaves independent internal permutations of the two colors. We will trim their large tuple marginals and combine them with inverse-image information from the first block.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Use the ambient graph of all hypothetical curvature tags satisfying the fixed basic bounds, not just the currently present vertices.” · Show in paper
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training· Method
PaperSource excerpt
“Each communication tensor is partitioned into contiguous micro-groups of G elements, where G ∈ 32, 64, 128, 256, 512. Each group is processed independently to capture localized statistics, enabling fine-grained and adaptive scaling while preserving high-efficiency GPU execution.” · Show in paper
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Weight tying yields a substantial improvement, especially in the low-budget regime, consistent with its role in making retention scores comparable across layers and heads.” · Show in paper
Verified
Table 4 directly shows a +11.51 point gain at budget 256 and +6.93 point gain at budget 512 from weight tying.
Common dyadic time-tree dual square summation for frequency summation without logarithmic loss.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“The argument below is the common-tree mechanism of the planar companion [24, Section 9]. We give the full calculation in dimension n: its spatial rescaling is the point at which the endpoint exponent enters.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Theorem 4.1 (Refinement). Use the parameters and interval tree of Section 2 in the large-input branch. Let A be a power of two with 1 ≤ A ≤ Ainit, and let U be an integer table on every labeled tree node and every valid target state, with 0 ≤ U(I, q) ≤ N, c(I, q) ≤ U(I, q) ≤ Ac(I, q). Set a = max{1, A/F}. There is a randomized, finitely executable refinement table dS such that, for every state (I, q), dS(I, q) ≥ c(I, q)/a, E[(dS(I, q) − c(I, q))+] ≤ c(I, q)/ℓ. (4.1)” · Show in paper
Verified
The proof carefully clips intermediate underestimates at true cost in the analysis so they cannot cancel errors elsewhere, and all numerical bounds are verified explicitly.
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
PaperSource excerpt
“In addition, FP4Quant exhibits clear sensitivity to group size, where increasing G leads to gradual degradation in TP and a sharp breakdown at large G (e.g., G = 512) in both TP and PP.” · Show in paper
Verified
Table 2 and Figure 5 clearly show the progressive degradation in TP and the sharp breakdown at G = 512 in both TP and PP.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56× speedup and 2× memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node.” · Show in paper
Verified
The experimental results across accuracy benchmarks (Tables 3–7, 9) and system benchmarks (Figures 8–9, Tables 8, 10) directly support the claimed speedup, memory reduction, model coverage, and single-node 530B serving.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
PaperSource excerpt
“Despite the advantages granted to the baseline methods, TurboQuant consistently outperforms both Product Quantization and RabitQ in terms of recall ratio across all experiments.” · Show in paper
Verified
Table 2 and Figure 5 clearly demonstrate both the orders-of-magnitude speedup in quantization time and the superior recall@1@k across all tested datasets and bit-widths.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“As shown in Figure 9 (top), compared to FasterTransformer’s FP16 implementation of OPT, SmoothQuant-O3 can further reduce the execution latency of OPT-13B and OPT-30B by up to 1.56× when using a single GPU. This is challenging since FasterTransformer is already more than 3× faster compared to the PyTorch implementation for OPT-30B. Remarkably, for bigger models that have to be distributed across multiple GPUs, SmoothQuant achieves similar or even better latency using only half the number of GPUs (1 GPU instead of 2 for OPT-66B, 4 GPUs instead of 8 for OPT-175B).” · Show in paper
Verified
Figures 8 and 9 and Table 8 directly report the measured speedups, memory reductions, and GPU count reductions.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“The rectangular coordinates below make this action transitive on point and ordered-line columns. Target affine maps commute with it, so neighbor updates remain column permutations after row encoding.” · Show in paper