By Md Romyull IslamSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2608-23816.git
Cite
Version link
https://openversions.com/index/arxiv-2608-23816/commit/adb95be4dc252dcf3dca4ce3f7b22859d9b1c077
Version identifier (commit hash)
adb95be4dc252dcf3dca4ce3f7b22859d9b1c077

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

AQLoRA is a zero-search, data-free recipe that uses one CPU pass over pretrained weights to set per-layer precision and adapter placement, achieving speedups over well-tuned QLoRA (11.3 ± 2.3% faster in its speed setting with 0.1 to 1.5 accuracy points lost, and 4.8 ± 2.0% faster in its quality setting at accuracy parity with QLoRA and within a point of fp16 LoRA) across six models and four architecture families (1.4B to 14B) on Commonsense-170K, for a memory premium of 0.06 to 0.8 GiB on 1–4B models, 0.5 GiB at 7B, and 1.9 GiB at 14B.

Across thirteen independent timing sessions on five models, AQLoRA-s averages 11.3 ± 2.3% faster than well-tuned QLoRA and is faster in every session (by 7.0% to 15.2%), while giving up 0.1 to 1.5 accuracy points.

Across thirteen independent timing sessions on five models, AQLoRA-q averages 4.8 ± 2.0% faster than well-tuned QLoRA (2.1% to 8.1% faster in every session), while matching QLoRA accuracy on every model and staying within 1 point of fp16 LoRA on Commonsense-170K and GSM8K.

Choosing protected layers by NF4 reconstruction error (εℓ\varepsilon_\ell) is not measurably better than parameter-matched random selection on base perplexity (across five models and three budgets, p=0.12p=0.12) or downstream accuracy (tested on phi-1.5 and Phi-3-mini); the number of protected layers drives the speed, not which layers are protected.

Choosing adapter layers by weight density (sℓs_\ell) is no better than random selection, with random selection outperforming density in all four tested cells on phi-1.5 at α∈{0.5,0.7}\alpha \in \{0.5, 0.7\}.

AQLoRA's NF4-error-based precision map matches the downstream accuracy of Unsloth's hand-curated dynamic-4bit selection on Qwen2.5-7B (89.0 vs. 89.0 on the 8-task average) while requiring only a 5 to 37 s CPU pass and no per-model curation.

The CPU profiling pass takes 5 s on 1B models, 12 to 13 s at 3 to 4B, 25 s at 7B, and 37 s at 14B on 40 CPU threads with the checkpoint in the page cache.

A noise floor measured within a single timing session understates the spread between independent sessions (e.g., on Phi-3-mini, a 1.3% within-session floor against an 8.2-point span across three kept sessions), and random seeds control almost none of the run-to-run variance.

Methods

CPU profiling pass computing per-layer NF4 reconstruction MSE εℓ=1∣Wℓ∣∑b∥b−absmax-NF4−1(absmax-NF4(b))∥22\varepsilon_\ell = \frac{1}{|W_\ell|} \sum_b \| b - \text{absmax-NF4}^{-1}(\text{absmax-NF4}(b)) \|_2^2 and weight density sℓ=Pr⁡[∣Wℓ,ij∣<τ]s_\ell = \Pr[|W_{\ell,ij}| < \tau] with τ=0.01\tau=0.01 and block size 64 over the fp16 checkpoint without data or forward passes.

Source excerpts

“For each linear layer ℓ with weights Wℓ, flattened into blocks b of 64 elements (matching bitsandbytes), εℓ = 1/|Wℓ| ∑_b ||b − absmax-NF4^-1(absmax-NF4(b))||_2^2, sℓ = Pr[|Wℓ,ij| < τ], with τ=0.01 and the true 16-level NF4 quantile grid. The pass runs on CPU over the fp16 checkpoint, is data-free, and requires no forward pass.” · Show in paper

Zero-search precision allocation keeping the top K=⌈ρL⌉K = \lceil \rho L \rceil linear layers with largest εℓ\varepsilon_\ell (plus lm_head) in fp16 and quantizing all other linear layers to NF4 with double quantization.

Source excerpts

“Given a protection budget ρ ∈ [0, 1] (sensitive-frac), the K=⌈ρL⌉ layers with largest εℓ are loaded in fp16; all others in NF4 with double quantization; lm_head stays fp16.” · Show in paper

Adapter placement strategies: AQLoRA-q adapts every layer (α=1.0\alpha=1.0) for quality, while AQLoRA-s uses depth placement on the top blocks (α=0.85\alpha=0.85) so autograd stops the backward pass early.

Source excerpts

“AQLoRA-q is the quality setting: adapters on every layer (α=1.0). AQLoRA-s is the speed setting: depth placement with α=.85 and early backward stopping.” · Show in paper

Fair-lever evaluation and shared-hardware timing protocol: applying SDPA attention, LoRA dropout 0, length-grouped dynamic batching, and fused AdamW to all baselines, timing methods interleaved on one GPU over a fixed measurement duration, estimating speed as the mean of the fastest half of repeats, and excluding sessions with a noise floor of 3% or more.

Source excerpts

“All experiments also use a set of shared levers, applied to every method: SDPA attention, LoRA dropout 0, length-grouped dynamic batching, and a fused AdamW.” · Show in paper

“We therefore estimate speed as the mean of the fastest half of the repeats: three or four runs rather than one lucky maximum, without the interference-slowed tail. ... A session is kept when its floor is under 3%; at or above that, the floor is the size of the effect being measured.” · Show in paper