Headline finding
JoLT achieves near-lossless KV-cache compression (2-3x compression with less than 0.2% perplexity degradation) without retraining across diverse model architectures, and this free zone carries over to downstream reasoning and long-context retrieval.
The experimental results in Table 1, Tables 4-7, and Table 9 support the 2-3x near-lossless claim across the tested models and tasks, noting that LLaMA-3.1-8B and Qwen2.5-14B used FlashJoLT.
“Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2–3× compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3× and declines by only 0.90 and 2.40 pp at 4× and 5×, respectively.” · Show in paper