By Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris GinsburgSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2404-06654.git
Cite
Version link
https://openversions.com/index/arxiv-2404-06654/commit/444e346ed7caf8478b63a8a65328a8ab0dab3c99
Version identifier (commit hash)
444e346ed7caf8478b63a8a65328a8ab0dab3c99

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2404.06654v3 OpenVersions: 1b40ed92-de74-4ccb-9a0e-034922762387

Posted

Claims

Findings

Headline finding

Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all of the 17 evaluated long-context LMs exhibit large performance drops on RULER as context length increases, and while all claim context sizes of 32K tokens or greater, only about half of them maintain satisfactory performance at 32K.

Gemini-1.5-Pro and GPT-4 consistently outperform all other evaluated models on RULER regardless of whether the weighted average ranking scheme emphasizes longer sequences (wAvg. inc) or shorter sequences (wAvg. dec).

Tasks in each of the four categories of RULER (NIAH, VT, AG, QA) form cohesive clusters of their own without redundancy.

Training on longer sequences does not always lead to better performance on RULER, and a large training context window is not always necessary for good long-context performance.

Larger model sizes positively correlate with better long-context capabilities on RULER, improving both short-context (4K) performance and relative degradation as context length increases.

Non-Transformer architectures, specifically RWKV-v5 and Mamba-2.8B-slimpj, lag behind the Transformer baseline Llama2-7B by large margins on RULER and degrade significantly when extending context size to 8K.

Yi-34B-200K exhibits large performance degradation on RULER as input length (up to 256K) and task complexity increase, failing due to non-robustness to needle types, inability to ignore distractors, returning incomplete information, verbatim copying from context, unreliable entity tracking, inaccurate aggregation, and frequent hallucination in long-context QA.

Methods

Evaluate 17 aligned long-context LLMs (and 7 base models) using vLLM in BFloat16 on 8 NVIDIA A100 GPUs with greedy decoding across 13 representative RULER tasks, generating 500 examples per task at context lengths of 4K, 8K, 16K, 32K, 64K, and 128K with model-specific chat templates and answer prefixes.

Source excerpts

“We evaluate all models using vLLM (Kwon et al., 2023), an LLM serving system with efficient KV cache memory management. For all models, we run the inference in BFloat16 on 8 NVIDIA A100 GPUs with greedy decoding.” · Show in paper

“For each task, we evaluate each model with 500 examples generated for each length from the series (4K, 8K, 16K, 32K, 64K, 128K), while complying with each model's necessary chat template.” · Show in paper

Determine effective context length as the maximum evaluated length where a model's average RULER score exceeds the Llama2-7B 4K baseline (85.6%), and rank models using two weighted average scores (wAvg. inc and wAvg. dec) with linearly increasing and decreasing weights across context lengths.

Source excerpts

“We use the performance of Llama2-7b model at the 4K context length as the threshold. We report in Table 3 the maximum length exceeding the threshold as the "effective length" along with the "claimed length".” · Show in paper

“We rank models under two weighting schemes: wAvg. (inc) and wAvg. (dec) where the weight linearly increases and decreases with sequence length respectively.” · Show in paper

Select 13 representative tasks from 18 task configurations by evaluating 8 open-source models across context sizes and applying agglomerative clustering with correlation coefficient as the distance metric to remove redundant tasks within clusters.

Source excerpts

“We evaluate eight open-sourced models at various context sizes across 18 task configurations. Each task can then be represented with a vector of model performance at various context sizes. The 18 task vectors are then clustered via agglomorative clustering algorithm, using correlation coefficient as the distance metric.” · Show in paper

Generate synthetic long-context tasks across four categories: Retrieval (S-NIAH, MK-NIAH, MV-NIAH, MQ-NIAH with words, numbers, or UUIDs in noise sentences or Paul Graham essays), Multi-hop Tracing (Variable Tracking via linear chains of variable bindings), Aggregation (CWE using uniform distributions and FWE using Zeta distributions), and QA (SQuAD and HotpotQA augmented with distracting paragraphs).

Source excerpts

“Based on these criteria, we develop four NIAH tasks. The "needle" in each of these tasks is a key-value pair inserted into the "haystack" (long distractor texts).” · Show in paper

“In the common word extraction task (CWE), words are sampled from discrete uniform distributions, with the number of common words fixed while the number of uncommon words increases with the sequence length. In the frequent words extraction task (FWE), words are sampled from Zeta distribution.” · Show in paper

“In this task category, we insert the golden paragraphs (i.e., the paragraphs that contain answers) into paragraphs randomly sampled from the same dataset.” · Show in paper