Headline finding
QATFactory is an open-source, deployment-aligned framework for quantization-aware distillation (QAD) and reinforcement learning (QARL) that simulates deployment-time quantization in BF16 across NVFP4, MXFP4, and Q4_K formats, dense and MoE models, and full-parameter or LoRA training, exporting checkpoints directly to vLLM, SGLang, and llama.cpp without additional lossy conversion or inference overhead.
The manuscript thoroughly documents the framework design, supported formats, training workflows, and direct evaluation of exported checkpoints in vLLM and llama.cpp.
“We present QATFactory, an open-source framework for deployment-aligned quantization-aware distillation (QAD) and reinforcement learning (QARL). QATFactory simulates deployment-time quantization while performing matrix multiplications in BF16, allowing models to adapt to quantization noise without requiring training hardware that natively supports the target format; for example, it supports NVFP4 training on H100 GPUs, which lack FP4 Tensor Cores. The framework supports NVFP4, MXFP4, and llama.cpp’s Q4_K format; dense and mixture-of-experts models; and both full-parameter and LoRA-based training. It exports checkpoints directly to vLLM and llama.cpp without an additional lossy conversion step or added inference overhead.” · Show in paper