Headline finding
RocketKV provides a compression ratio of up to 400x, end-to-end speedup of up to 3.7x, and peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.
The reported 400x compression ratio, 3.7x A100 speedup, 32.6% peak memory reduction, and near-Full-KV accuracy are directly supported by the experimental results in Section 4.2, Section 4.3, Figure 5, Figure 7, and Table 2.
“We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.” · Show in paper