By Payman Behnam, Debopam Sanyal, Pranav Ponnusamy, Kusum Meda Ravi, Pranav Sastry, Mani Chandana Bandaru, Alind Khare, Myungjin Lee, Alexey TumanovSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2611-11761.git
Cite
Version link
https://openversions.com/index/arxiv-2611-11761/commit/b7142992f86eb981095b7e924aa245e28d95d590
Version identifier (commit hash)
b7142992f86eb981095b7e924aa245e28d95d590

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

BridgeKV enables large-model prefill and small-model decoding within a single request via a learned KV-stitching module, outperforming parameter-matched fine-tuned small-model baselines across all 30 settings (six model pairs and five datasets), closing up to 80% of the quality gap to the larger model, achieving 3.0x faster requests on one A40 GPU for Llama-3.1-Instruct-8B->Llama-3.2-Instruct-1B, and predicting up to 33.86x faster requests and 27.5x lower energy use in serving simulations for Llama-3.1-Instruct-70B->Llama-3.2-Instruct-1B.

Across six model pairs (Llama and Qwen families) and five datasets, BridgeKV improves test accuracy over the parameter-matched fine-tuned decode anchor in all 30 settings by up to 8.4% relative (and at least 0.2% relative in the worst case), closing up to 80% of the gap to the larger prefill anchor.

In 12 of the 30 evaluated settings, BridgeKV uses only the first training stage so the decode anchor decodes with the exact weights of its fine-tuned baseline, and BridgeKV still scores higher in all 12 settings.

Training the KV-stitching module is cheaper in A40 GPU-hours than training the parameter-matched PEFT baseline in every evaluated setting.

On one NVIDIA A40 GPU at batch size 1, a batched implementation of the KV-stitching module adds 1.6-6.7% to the prefill time of the prefill anchor across the four 1B/0.5B-decode pairs, and Llama-3.1-Instruct-8B->Llama-3.2-Instruct-1B serves a 1K-token prompt and 256-output-token request in 2.96 s (3.0x faster than 8B alone at 8.83 s, and close to 1B alone at 2.80 s).

In a Splitwise-style discrete-event serving simulation (1024 input/1024 output tokens, 200 requests, 2.0 req/s), BridgeKV with Llama-3.1-Instruct-8B->Llama-3.2-Instruct-1B reduces response time by 4.36x (39.80 s to 9.13 s), increases throughput by 4.40x (25.81 to 113.5 tok/s), and reduces energy per request by 4.31x (11.98 Wh to 2.78 Wh) compared to Llama-3.1-Instruct-8B alone.

In the serving simulation, BridgeKV's benefits scale with the model-size gap between prefill and decode anchors, reaching up to 33.86x response speedup and 27.5x energy savings for Llama-3.1-Instruct-70B->Llama-3.2-Instruct-1B.

Weighted-layer mapping outperforms static layer mapping policies (ordered, shifted, and grouped mapping) on NarrativeQA, and residual stitching outperforms linear, MLP, convolutional, and bottleneck stitching on NarrativeQA.

Methods

Windowed layer mapping assigns each decode layer j to a local window of W consecutive prefill layers centered on relative depth cj=1+⌊(j−1)(Lp−1)max⁡(1,Ld−1)⌋c_j = 1 + \lfloor \frac{(j-1)(L_p-1)}{\max(1, L_d-1)} \rfloor and combines them either via a learned softmax-weighted sum with separate key/value logits or via feature concatenation.

Source excerpts

“For each decode layer j ∈ {1, ..., Ld}, BridgeKV first finds the prefill layer at the same relative depth, cj = 1 + floor((j-1)(Lp-1)/max(1, Ld-1)), and takes the window of W consecutive prefill layers centered on cj, shifted inward at the first and last layers so that it always contains W valid layers” · Show in paper

KV cache stitching applies separate key and value residual nonlinear layers Gψ,j(z)=Pjzˉ+bj+αjDjσ(Ujzˉ)G_{\psi,j}(z) = P_j \bar{z} + b_j + \alpha_j D_j \sigma(U_j \bar{z}) with zˉ=LN(z)\bar{z} = \mathrm{LN}(z), undoing MpM_p's RoPE before the key map and applying MdM_d's RoPE afterward, and initializes Pj,bjP_j, b_j in closed form via regularized least squares.

Source excerpts

“Gψ,j(z) = Pj z̄ + bj + αj Dj σ(Uj z̄), z̄ = LN(z)” · Show in paper

“To avoid this position dependence, we undo the RoPE of Mp on the source keys, apply the key stitching layer, and then apply the RoPE of Md” · Show in paper

“We initialize the linear projection of every stitching layer in closed form. We run both anchors on 512 training examples and sample up to 256 positions from each.” · Show in paper

Two-stage training trains only ψ\psi in Stage 1 with both anchors frozen using Lalign(ψ)+λKLLKL(ψ)+λrecLrec(ψ)\mathcal{L}_{\text{align}}(\psi) + \lambda_{\text{KL}} \mathcal{L}_{\text{KL}}(\psi) + \lambda_{\text{rec}} \mathcal{L}_{\text{rec}}(\psi) (MSE + cosine cache alignment and next-token KL distillation), and optionally trains ψ\psi with LoRA adapters η\eta on MdM_d in Stage 2 using teacher-forced cross-entropy Ltask(ψ,η)\mathcal{L}_{\text{task}}(\psi, \eta).

Source excerpts

“The first stage trains only ψ, with both anchors frozen, and combines two losses: an alignment loss that keeps the stitched cache close to the native cache of Md, and a distillation loss that makes Md, reading the stitched cache, match the next-token distribution of Mp.” · Show in paper

“In the second stage, we train ψ together with low-rank adapters (LoRA) (Hu et al., 2022) η on Md, using the teacher-forced cross entropy of Md on the stitched cache” · Show in paper

Batched KV-stitching implementation stacks weights across all LdL_d decode layers to execute RoPE un-rotation, window aggregation, LayerNorm, three batched matrix multiplications for Eq. (6), and RoPE re-application in a single pass compiled with torch.compile.

Source excerpts

“All stitching layers of a pair have the same shapes, so the batched implementation stacks their weights and runs the whole module in one pass.” · Show in paper

“evaluates Eq. (6) for all layers with three batched matrix multiplications, and applies the RoPE of Md; we compile the pass with torch.compile (Ansel et al., 2024).” · Show in paper

System-level performance is evaluated using a Splitwise-style discrete-event simulator modeling 8 NVIDIA A40 GPUs, exponential request arrivals (2.0 req/s over 200 requests), layer-wise overlapped KV-cache transfer over NVLink (100 GB/s) or InfiniBand (50 GB/s), and phase-specific power (95% TDP for prefill, 45% TDP for decode).

Source excerpts

“Unless otherwise stated, the main system results use a 200-request trace with 1024 input tokens and 1024 output tokens per request. Request arrivals follow an exponential distribution with an arrival rate of 2.0 requests/s.” · Show in paper

“For energy calculation, we use the phase-specific power model from Splitwise (Patel et al., 2024). The prompt phase is modeled as compute-bound and uses 95% of machine TDP, while decode is modeled as memory-bandwidth-bound and uses 45% of aggregated GPU TDP” · Show in paper