By Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel IzmailovSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2606-09659.git
Cite
Version link
https://openversions.com/index/arxiv-2606-09659/commit/4deb725f99ac745b39e6e76fb037e05a51aa41eb
Version identifier (commit hash)
4deb725f99ac745b39e6e76fb037e05a51aa41eb

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Latent Context Language Models (LCLMs)—continually pre-trained 0.6B-encoder, 4B-decoder compressors at compression ratios of 1:4, 1:8, and 1:16—establish a new Pareto frontier across general-task performance, compression speed (TTFT), and peak memory usage compared to KV cache compression baselines.

LCLMs are faster and more accurate than smaller uncompressed language models (Qwen3 and Llama 3 models from 0.6B to 8B parameters); at 64k tokens, the 16x LCLM reaches the first token in 0.21 s (vs. 0.81 s for Llama 3.2 1B and 0.97 s for Qwen3-0.6B) while achieving 66.25% accuracy on LongHealth5 (vs. 22.50% and 32.75%), and the 4x LCLM exceeds uncompressed Qwen3-4B-Instruct-2507 on LongBench with a 6x lower TTFT.

LCLMs achieve the fastest compression time at every context length from 4K to 1M tokens and substantially lower peak GPU memory at long contexts, where all KV cache compression baselines run out of memory at 512K tokens (except Attention Matching, which fails at 512K due to numerical instability and runs out of memory at 1M), while the 16x LCLM grows from 9 GB at 4K to 23 GB at 512K and 37 GB at 1M tokens.

Controlled from-scratch pre-training architecture search and large-scale validation show that an encoder window size W=1024W = 1024, causal masking, an MLP-only adapter, no boundary overlap, and an embedding-initialized encoder yield the lowest pre-training loss and strongest overall performance, with mean pooling preferred at 16x compression and concatenation slightly outperforming mean pooling at 4x compression.

Equipping the 16x LCLM with a segment-expansion tool that retrieves uncompressed text on demand substantially improves retrieval accuracy on RULER needle-in-a-haystack tasks compared to the same 16x LCLM without expansion (improving average accuracy from 80.94% to 99.44% at 4k, 75.88% to 99.06% at 8k, and 69.56% to 95.06% at 16k).

Online latent compaction—iteratively encoding the model's own generated chain-of-thought tokens into 16x compressed latent memory during decoding—is nearly lossless on the full MATH test set (90.52% vs. 92.46% accuracy for full-context generation) while reducing average KV length by 3.1x, average per-problem peak KV by 3.6x, and maximum peak KV by 7.9x (from 32,768 to 4,135 tokens).

LCLMs can compress short, information-dense contexts (GSM8K) with high accuracy, achieving the highest exact-match accuracy across all compression ratios (89.23% at 4x, 87.79% at 8x, 81.27% at 16x) and outperforming KV cache compression baselines, especially at higher compression ratios.

A multi-stage training recipe (adapter warmup, encoder training, end-to-end continual pre-training, and SFT) prevents model degradation compared to direct end-to-end training from the beginning, and full-parameter training outperforms frozen-decoder or LoRA variants, with minimal drift from the original decoder when compression is disabled.

Increasing the size of the decoder (from 4B to 8B) reduces stage-2 pre-training loss much more than increasing the size of the encoder (from 0.6B to 4B), although downstream evaluation results are mixed.

Methods

LCLM architecture: an encoder splits the input sequence into windows of size WW, computes hidden states, aggregates every NN tokens via a pooling operator (mean pooling or concatenation) into latent tokens, and projects them to the decoder's embedding dimension via an MLP adapter.

Source excerpts

“An LCLM consists of an encoder with a pooling method that maps token chunks to soft tokens, an adapter to project to the decoder's embedding dimension, and a decoder that consumes the soft tokens as latent context.” · Show in paper

“Because the encoder and decoder may have different hidden dimensions, an adapter a(·) projects the compressed latent sequence z1:M from the encoder hidden dimension into the decoder hidden dimension, producing latent tokens s1:M.” · Show in paper

Multi-stage training recipe: Stage 0 (adapter warmup with frozen encoder and decoder), Stage 1 (encoder training with frozen decoder), Stage 2 (end-to-end continual pre-training on interleaved compressed/uncompressed spans and auxiliary reconstruction), and Stage 3 (supervised fine-tuning on reasoning, long-context QA, general instruction, and agentic trajectories).

Source excerpts

“Stage 0: Adapter warmup. We freeze both the encoder and decoder and update only the adapter. • Stage 1: Encoder training. We unfreeze the encoder while keeping the decoder frozen. • Stage 2: End-to-end continual pre-training. We unfreeze the decoder with a small learning rate. • Stage 3: Supervised fine-tuning. We train on the SFT mixture and increase the decoder learning rate.” · Show in paper

Agentic segment expansion: segmenting long inputs into fixed-size chunks of 1,024 encoder tokens, compressing each chunk with an assigned segment identifier, and equipping the LCLM with an `expand` tool that returns the original uncompressed text of a requested segment on demand.

Source excerpts

“We segment the input into fixed-size chunks of 1,024 encoder tokens, compress each chunk, and assign it a segment identifier. The model receives the entire compressed sequence in a single prompt, together with one expand tool. At each turn, the model can issue a native tool call to expand with a segment identifier (Figure 6), and the tool returns the original text of that segment.” · Show in paper

Online latent compaction: after the first 768 generated tokens, encoding the oldest 512 generated tokens into 32 latent tokens (16x compression) while keeping the prompt and most recent 256 tokens uncompressed, and repeating after every further 512 generated tokens.

Source excerpts

“After the first 768 generated tokens, we encode the oldest 512 generated tokens into latent memory tokens at a 16x compression ratio, resulting in 32 latent tokens, and keep the most recent 256 tokens uncompressed. Generation then continues from the compacted context and latest raw tokens, and the procedure repeats after every further 512 generated tokens.” · Show in paper