By Yuchen Yang, Yifan Zhao, Anisha Dasgupta, Sasa MisailovicSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-16184.git
Cite
Version link
https://openversions.com/index/arxiv-2607-16184/commit/3754f754649b482b987a5f0d0c6740ba8e884bac
Version identifier (commit hash)
3754f754649b482b987a5f0d0c6740ba8e884bac

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over existing quantization baselines, achieving FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94× throughput improvement, and improving quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.

Across Qwen1.5-MoE-A2.7B, Mixtral-8×7B-v0.1, and Gemma-4-26B-A4B, PagedWeight consistently achieves a better quality-memory tradeoff than APL, DP-LLM, and MxMoE on perplexity (Wikitext2, C4) and reasoning accuracy (GSM8K, MATH-500) tasks.

Under large KV-cache demand on Qwen1.5-MoE-A2.7B, PagedWeight preserves long-context quality on LongBench tasks (Passage Retrieval, NarrativeQA, QMSum) with much lower memory consumption than FP16 and outperforms APL at similar memory budgets, with the largest gains under tight memory.

PagedWeight maintains generation throughput close to the uniform APL-style baseline at similar or lower GPU memory consumption on Qwen1.5-MoE-A2.7B (sequence length 2048, batch sizes 1 and 4), with at most a 3.3% throughput drop at batch size 1 and 4.1% at batch size 4.

Each component of PagedWeight—offline global sensitivity, online routing statistics, prompt residuals, and asynchronous page movement—contributes to lowering perplexity on Qwen1.5-MoE-A2.7B (Wikitext2 and C4).

Methods

PagedWeight represents each expert linear-block in an Any-Precision (AP) bit-plane and lookup table (LUT) format organized in a weight page table, and asynchronously offloads or reloads weight pages between GPU memory and CPU RAM at safe boundaries.

Source excerpts

“PagedWeight quantizes a network at the granularity of per expert linear-block (a weight in an expert). Each linear-block is identified by a triple index: i := (ℓ, e, u) for layer ℓ, expert e, and one of two linear-block types u ∈ {gate_up, down}. PagedWeight organizes linear-blocks and their quantization decision in a weight page table, where each linear-block i occupies one row.” · Show in paper

“For offload, the system copies the weight page to CPU RAM and commits the lower qi. Once the transition is safe, the system retires the GPU pages above the committed precision, and the released allocator headroom can be used by the KV blocks. For reload, the runtime restores the offloaded weight pages first and commits the higher qi only after the page state is GPU-resident.” · Show in paper

PagedWeight's quality-aware runtime planner combines offline Hessian-based global sensitivity, online routing-mass bucketing (with damage multipliers and bitwidth floors), and runtime prompt residuals predicted by linear regression heads on routed-input norm features to greedily select bitwidth reductions sorted by predicted damage per released byte until the KV-cache-derived byte target is met.

Source excerpts

“For each action, the planner estimates accuracy impact by combining three factors: offline global sensitivity, online routing statistics, and runtime prompt residual.” · Show in paper

“The planner considers each legal bitwidth reduction as one step. It sorts the steps by predicted damage per released byte, d̂/∆, and greedily takes the cheapest steps until the target D is met.” · Show in paper

PagedWeight implements a fused mixed-precision MoE CUDA kernel that reads Any-Precision bit-planes and LUTs directly at linear-block granularity and performs routing, expert activation, down projection, and output accumulation in a single kernel.

Source excerpts

“PagedWeight provides a fused mixed-precision MoE AP kernel to enable per-expert bitwidths at runtime.” · Show in paper

“The kernel reads weight pages directly and enables different bitwidths for each linear-block, and does the routing, expert activation, down projection, and output accumulation in one fused CUDA kernel.” · Show in paper