Headline finding
Latent Context Language Models (LCLMs)—continually pre-trained 0.6B-encoder, 4B-decoder compressors at compression ratios of 1:4, 1:8, and 1:16—establish a new Pareto frontier across general-task performance, compression speed (TTFT), and peak memory usage compared to KV cache compression baselines.
The accuracy, TTFT, and peak memory benchmarks across RULER, LongBench, LongHealth5, and GSM8K support the claim that LCLMs improve the Pareto frontier over KV cache compression baselines.
“We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage.” · Show in paper