By Estelle Zheng, Sébastien Warichet, Emmanuel Helbert, Christophe CerisaraSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2608-25677.git
Cite
Version link
https://openversions.com/index/arxiv-2608-25677/commit/b5f74122c70b7f08b2c9505ef4d0a59fe3be6993
Version identifier (commit hash)
b5f74122c70b7f08b2c9505ef4d0a59fe3be6993

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

In factual acquisition on an anonymized OpenStreetMap-derived benchmark with Qwen3-4B, QLoRA rank (8, 16, 32, 64) induces an acquisition-retention frontier: low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, higher ranks improve same-fact paraphrase generalization at an increasing cost to unrelated benchmarks, and full fine-tuning (FFT) behaves as a conservative baseline that retains general capabilities well without reaching the highest factual-acquisition regime.

OOD degradation with increasing QLoRA rank is broad across HumanEval, IFEval, MMLU-Redux, and BBH, whereas TruthfulQA remains comparatively stable.

An unquantized standard-LoRA rank sweep on Qwen3-1.7B (ranks 8, 16, 32) exhibits the same qualitative monotonic acquisition-retention trade-off, suggesting that quantization is not required for the qualitative rank trend.

When checkpoints are matched to target paraphrase accuracies (75%, 85%, 95%), higher-rank QLoRA configurations achieve stronger paraphrase performance only with larger OOD losses, suggesting that the apparent robustness of low-rank adaptation to forgetting is partly due to limited plasticity.

Model-drift diagnostics mirror the behavioral acquisition-retention trade-off: higher-rank QLoRA checkpoints exhibit larger symmetric KL divergence from the pretrained model, larger effective dense update magnitudes (RMS drift), and larger SVD intruder excess than lower-rank QLoRA and FFT.

In a separate math reasoning adaptation experiment on OpenR1-Math-220k, FFT and QLoRA (r = 16, 32) obtain nearly identical average math performance and small OOD drops, indicating that the strong rank-dependent frontier is most pronounced when adaptation installs new factual associations rather than reinforcing skills already supported by pretraining.

Anonymizing entity names in the OpenStreetMap dataset sharply reduces base-model exact-match accuracy, gold-answer preference, and log-probability margins, confirming that the base model cannot solve the anonymized task directly from prior knowledge.

Methods

Construct an anonymized OpenStreetMap (OSM) factual acquisition benchmark from 14 small-city extracts (populations 5,000–80,000) with five atomic relation types, replacing all entity names with synthetic identifiers, yielding 1,938 training examples and 900 held-out paraphrase evaluation examples with disjoint surface templates.

Source excerpts

“We derive atomic facts from 14 city-level OSM extracts, linking entities (POIs, roads, and cities) to five relation types: POI category, containing city, nearest road, nearest POI, and road-length bucket. The training split contains 1,938 instruction-style question-answer examples... Evaluation uses 900 held-out examples derived from the same facts but expressed with disjoint surface templates” · Show in paper

“Since current LLMs might have some prior knowledge of popular global cities, we focus on smaller cities with populations between 5,000 and 80,000. To further reduce the influence of prior knowledge, all names of cities, POIs, and roads are replaced by synthetic identifiers such as C-TRAIN-001, POI-TRAIN-000001, and ROAD-TRAIN-000001.” · Show in paper

Diagnose base-model prior knowledge on anonymized and non-anonymized splits using exact-match (EM) generation accuracy and a teacher-forced gold-vs-distractor preference score with five sampled distractors matching answer type and relation.

Source excerpts

“We evaluate both anonymized and non-anonymized versions of the data as a question-answering task, where the model is prompted to generate the gold answer. We report exact-match (EM) generation accuracy and a teacher-forced gold-vs-distractor preference score. For each example, we sample five distractors from other gold answers in the same split, matching both answer type and relation whenever possible.” · Show in paper

Fine-tune Qwen3-4B using full fine-tuning (FFT) and QLoRA at ranks r∈{8,16,32,64}r \in \{8, 16, 32, 64\} over five random seeds with autoregressive loss applied only to answer tokens, plus an unquantized standard-LoRA control on Qwen3-1.7B at ranks r∈{8,16,32}r \in \{8, 16, 32\}.

Source excerpts

“We use Qwen3-4B (Yang et al., 2025a) as the base model and compare full fine-tuning (FFT) with QLoRA adapters of rank r ∈ {8, 16, 32, 64}. Each training example consists of a question and its gold answer, with the autoregressive loss applied only to answer tokens. All runs are repeated over five random seeds.” · Show in paper

“To test whether the within-adapter rank trend persists without quantization, we additionally run standard LoRA on Qwen3-1.7B (Yang et al., 2025a) at ranks r ∈ {8, 16, 32}, using the same OSM task and OOD evaluation suite.” · Show in paper

Evaluate out-of-domain (OOD) retention using LM Evaluation Harness across five benchmarks (HumanEval, IFEval, TruthfulQA, MMLU-Redux-2.0, and BBH) and measure forgetting as ΔOOD=OODbase−OODadapted\Delta_{\text{OOD}} = \text{OOD}_{\text{base}} - \text{OOD}_{\text{adapted}}.

Source excerpts

“We use LM Evaluation Harness (Gao et al., 2024) on five benchmarks: HumanEval (Chen et al., 2021), IFEval (Zhou et al., 2023), TruthfulQA (Lin et al., 2022), MMLU-Redux-2.0 (Gema et al., 2025), and BBH (Suzgun et al., 2023)... We define forgetting as the drop in average OOD score relative to the base model: ∆OOD = OODbase − OODadapted.” · Show in paper

Measure model drift from the pretrained base model using symmetric KL divergence Dsym(p0,pθ)D_{\text{sym}}(p_0, p_\theta), RMS-normalized dense weight drift DRMSD_{\text{RMS}} in the merged update space, and SVD intruder excess at threshold ϵ=0.8\epsilon = 0.8 (k=10,K=64k=10, K=64).

Source excerpts

“we therefore measure drift using KL divergence from the base model (Shenfeld et al., 2026), teacher-forced negative log-likelihood on gold OSM answers, RMS-normalized dense weight drift, and SVD-based intruder dimensions (Glorot and Bengio, 2010; Shuttleworth et al., 2025).” · Show in paper

“In our implementation, the defaults are k = 10 and K = 64... We use the intruder rate at ϵ = 0.8 as the main SVD diagnostic.” · Show in paper

Conduct a separate math-adaptation experiment by fine-tuning Qwen3-4B for one epoch on a 94k-example subset of OpenR1-Math-220k using FFT and QLoRA (r∈{16,32}r \in \{16, 32\}), evaluating Pass@1 on MATH-500, AIME'24, AIME'25, AMC'23, Minerva Math, and OlympiadBench alongside the five-benchmark OOD suite.

Source excerpts

“We run a separate reasoning experiment on a 94k-example subset of OpenR1-Math-220k (Hugging Face, 2025) to test whether the OSM trend also appears in a larger skill-adaptation regime. We evaluate Pass@1 on MATH-500 (Hendrycks et al., 2021), AIME’24 and AIME’25 (Mathematical Association of America, 2024), AMC’23 (American Mathematics Competitions, 2023), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). OOD degradation uses the same five-benchmark average as the main experiment, relative to the base Qwen3-4B” · Show in paper