By Weili Xu, Jisen Li, Yuqing Jian, Chenxi Li, Zhizhou Sha, Yifan Yu, Qingyang Wu, Chenfeng Xu, Zhongzhu Zhou, Tianyi Zhang, Ben AthiwaratkunSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2609-39223.git
Cite
Version link
https://openversions.com/index/arxiv-2609-39223/commit/348e60dbff92fa15e5464387f0f54506be86d0ac
Version identifier (commit hash)
348e60dbff92fa15e5464387f0f54506be86d0ac

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2609.39223v2 OpenVersions: 1d28ba17-4db1-4ae4-a026-5171a5957492

Posted

Claims

Findings

Headline finding

QATFactory is an open-source, deployment-aligned framework for quantization-aware distillation (QAD) and reinforcement learning (QARL) that simulates deployment-time quantization in BF16 across NVFP4, MXFP4, and Q4_K formats, dense and MoE models, and full-parameter or LoRA training, exporting checkpoints directly to vLLM, SGLang, and llama.cpp without additional lossy conversion or inference overhead.

QATFactory converges stably across model families, parameter counts from 8B to 230B, and NVFP4, MXFP4, and Q4_K formats, and exported QAD checkpoints achieve higher average benchmark accuracy and lower mean KL divergence than RTN, GPTQ, and MSE-optimal PTQ baselines in vLLM and llama.cpp.

For W4A4 deployment, quantizing activations during training is not always optimal: W4A16 training produces higher-quality quantized checkpoints than W4A4 training for NVFP4, whereas MXFP4 benefits substantially from W4A4 training.

For Qwen3.5-9B at an 8K sequence length, rank-16 LoRA reduces GPU memory usage by 2.9x compared to full-parameter QAD, but LoRA underperforms full-parameter QAD and increasing the adapter rank does not close the quality gap.

Under a fixed training-token budget, QAD on fewer 32K sequences generalizes better than QAD on more 4K sequences, improving average accuracy across AIME25, BigCodeBench, and LiveCodeBench-v6 by 1.9 percentage points, and QAD improves or preserves quality across domains with the strongest transfer from code training to mathematical reasoning.

NVFP4 QARL reduces the rollout bottleneck and accelerates end-to-end RL training by 1.23x while improving mean accuracy across five mathematical reasoning benchmarks by 2.7 percentage points compared with BF16 RL followed by post-training quantization.

Methods

Deployment-aligned quantization-aware layers simulate target-format quantization and dequantization (Qκ\mathcal{Q}_\kappa and Qκ−1\mathcal{Q}_\kappa^{-1}) in BF16 during the forward pass and use a straight-through estimator (STE) to update continuous latent weights in the backward pass.

Source excerpts

“Let Qκ and Q−1 κ denote the quantization and dequantization operators for deployment configuration κ. Given a high-precision tensor X, the quantizer produces low-bit codes C and format-specific metadata θ: Qκ(X) → (C, θ), Q−1 κ (C, θ) → Xˆ .” · Show in paper

“Because discrete quantization has zero derivative almost everywhere, QATFactory uses a straight-through estimator (STE) during the backward pass, treating the quantize–dequantize transformation as an identity mapping” · Show in paper

Engine-specific scale sharing ties tensor-level weight scales and input-activation scales for fused projections (such as QKV in attention and gate/up in MLP) under NVFP4 to match vLLM and SGLang inference behavior.

Source excerpts

“In NVFP4, each fused matrix uses a single tensor-level weight scale and a common input-activation scale. To reproduce this inference-time behavior, QATFactory ties the corresponding scales during training: it computes the shared weight scale from the concatenated projection weights and tracks one activation scale for their common input.” · Show in paper

Quantization-aware distillation (QAD) minimizes token-averaged forward KL divergence between a frozen high-precision teacher and a quantized student initialized from the same weights, supporting full-parameter updates or LoRA-QAD (including per-expert sequential merging for MoE models).

Source excerpts

“the original high-precision pretrained checkpoint serves as a frozen teacher to supervise a low-precision student initialized from the same weights. Given an input sequence of length L, the framework minimizes the token-averaged forward Kullback–Leibler (KL) divergence between the teacher distribution pT,t and the student distribution pS,t over their shared vocabulary V” · Show in paper

“To control peak memory during LoRA-QAD, QATFactory merges and quantizes one expert weight (We + BeAe) at a time during the forward pass.” · Show in paper

Quantization-aware reinforcement learning (QARL) decouples rollout generation in vLLM using native low-precision kernels from policy training under simulated quantization, synchronizing packed low-precision weights after each step.

Source excerpts

“The rollout server loads the policy in the target low-bit format and uses an inference engine such as vLLM to generate responses with native low-precision kernels. Verifiers score these responses and send the resulting trajectories and rewards to the training server. The training server maintains high-precision latent weights and optimizer states in PyTorch.” · Show in paper

Engine-native checkpoint export merges trained LoRA adapters if present, quantizes the weights, and packs low-bit values, scale factors, and tracked activation scales directly into the target inference engine's native layout.

Source excerpts

“The exporter packs the low-bit values, scale factors, and format-specific metadata in the target engine’s native layout. It uses the same quantization configuration as training. When a format requires tracked state, such as NVFP4 input-activation scales, the exporter saves the values collected during training instead of running a separate calibration pass.” · Show in paper