By Mahendra Singh Rathor, Anagheem AzzamSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-25583.git
Cite
Version link
https://openversions.com/index/arxiv-2607-25583/commit/777db8529fa3b916973568089e3099570666af7b
Version identifier (commit hash)
777db8529fa3b916973568089e3099570666af7b

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

In the T5-small (60M-parameter) and WikiSQL setting, LoRA with r=16 on {q,v} is the Pareto-optimal configuration for unconstrained memory, recovering within 11.6 percentage points of full fine-tuning exact-match accuracy (59.6% vs. 71.2%, or 83.7% of full fine-tuning accuracy) while training fewer than 1% of parameters (589,824 parameters, 0.97%) and consuming 31% less peak GPU memory (1.60 GB vs. 2.31 GB).

Within the T5-small and WikiSQL setting, increasing LoRA rank from r=2 to r=16 produces consistent exact-match accuracy gains (from 38.6% to 59.6%), but rank beyond r=16 yields no measurable accuracy gain (r=32 achieves 60.4%, a 0.8-point gain despite doubling trainable parameters from 589,824 to 1,179,648), while peak memory remains nearly flat across all rank configurations (1.59–1.61 GB).

In the T5-small and WikiSQL setting, expanding adapted modules beyond {q,v} yields diminishing returns and is a less efficient lever than rank increase: expanding from {q,v} at r=8 (53.4%, 344K params) to {q,k,v,o} (59.2%, 590K params) or {q,v}+FFN (58.6%, 786K params) requires 71% or more additional parameters, whereas LoRA r=16 on {q,v} (59.6%, 590K params) matches {q,k,v,o} at the same parameter budget.

QLoRA with INT8 and NF4 quantization achieves comparable exact-match accuracy (52.8% and 53.2% respectively) to the FP16 LoRA r=8 baseline (53.4%) at dramatically lower peak training memory (0.60 GB for both, a 62% reduction relative to FP16 LoRA and a 74% reduction relative to full fine-tuning), with NF4 representing the best quantization trade-off due to lower inference latency (614 ms/query vs. 921 ms/query for INT8) and 34 tok/s throughput.

The rank ordering of configurations and Pareto front clustering are consistent across all three random seeds (42, 123, 456), while lower-rank configurations show higher variance (std up to 0.063 for r=2) and QLoRA configurations are among the most stable (std of 0.006 for INT8 and 0.015 for NF4).

Methods

Fine-tune T5-small (60M parameters) on 5,000 WikiSQL training examples and evaluate on 500 validation examples, serializing each example as 'translate English to SQL: <Q> table: <H1 | H2 | ...>' with the gold human_readable SQL string as target.

Source excerpts

“We fine-tune T5-small (60M parameters) for natural-language-to-SQL generation on WikiSQL. Each example is serialized as: translate English to SQL: <Q> table: <H1 | H2 | ···> and the target is the gold human_readable SQL string. We train on 5,000 examples and evaluate on 500 validation examples to maintain a tractable single-GPU sweep.” · Show in paper

Conduct a single-variable ablation study relative to the LoRA r=8, {q,v}, FP16 reference by varying LoRA rank r in {2, 4, 8, 16, 32}, adapted modules across {q,v}, {q,k,v,o}, and {q,v}+FFN, and numerical precision across FP16, INT8 (LLM.int8() mixed-precision), and NF4 4-bit (QLoRA with double quantization and FP16 compute dtype).

Source excerpts

“Table 1 lists all evaluated configurations; each changes one variable relative to the LoRA r=8, {q,v}, FP16 reference.” · Show in paper

“Following Dettmers et al. [2023] and Dettmers et al. [2022], we apply INT8 and NF4 4-bit quantization to the frozen base weights and attach LoRA adapters (r=8, {q,v}) on top. INT8 uses LLM.int8() mixed-precision; NF4 uses double quantization with FP16 compute dtype.” · Show in paper

Train all configurations using AdamW with learning rate 5e-4, batch size 8, 3 epochs, max input/output lengths of 256/128 tokens, beam search decoding (num_beams=4), LoRA dropout 0.1, and frozen biases across three random seeds (42, 123, 456) on a single NVIDIA T4 GPU (16 GB VRAM).

Source excerpts

“All configurations use AdamW, learning rate 5 × 10−4 , batch size 8, 3 epochs, max input/output lengths 256/128 tokens, and beam search decoding (num_beams=4). LoRA dropout is 0.1; biases are frozen. To assess variance, all configurations are run with three random seeds (42, 123, 456); mean and standard deviation are reported in Section 5.6.” · Show in paper

“All experiments run on a single NVIDIA T4 GPU (16 GB VRAM) via Kaggle Notebooks.” · Show in paper

Measure exact-match accuracy (case-insensitive, whitespace-normalized string comparison), execution accuracy (in-process SQLite engine over 25 executable examples), peak training memory via torch.cuda.max_memory_allocated, and single-query latency as the mean over 50 held-out examples after a warmup pass.

Source excerpts

“Exact-match accuracy: normalized string comparison between generated and gold SQL (case-insensitive, whitespace-normalized). Execution accuracy: agreement of query results when executed against the example table via an in-process SQLite engine, reported over the executable subset (25 of 500 evaluation examples).” · Show in paper

“Peak training memory is measured via torch.cuda.max_memory_allocated; single-query latency is the mean over 50 held-out examples after a warmup pass.” · Show in paper