Headline finding
BridgeKV enables large-model prefill and small-model decoding within a single request via a learned KV-stitching module, outperforming parameter-matched fine-tuned small-model baselines across all 30 settings (six model pairs and five datasets), closing up to 80% of the quality gap to the larger model, achieving 3.0x faster requests on one A40 GPU for Llama-3.1-Instruct-8B->Llama-3.2-Instruct-1B, and predicting up to 33.86x faster requests and 27.5x lower energy use in serving simulations for Llama-3.1-Instruct-70B->Llama-3.2-Instruct-1B.
All components of the overarching claim are directly supported by Figure 3, Figure 1, Section 4.3, and Table 7.
“Across six model pairs and five datasets, BridgeKV outperforms fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings. It closes up to 80% of the quality gap to using the larger model for both phases. On one A40 GPU, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B achieves 3.0× faster requests than the larger model alone, with latency close to the smaller model alone. BridgeKV can improve the hardware and system solutions that separate prefill and decode phases across GPUs. A separate serving simulation models request arrivals, queues, and cache transfer. For Llama-3.1-Instruct-70B→Llama-3.2-Instruct-1B, it predicts up to 33.86× faster requests and 27.5× lower energy use than using the larger model for both prefill and decode phases.” · Show in paper