Headline finding
In multilingual settings, Llama-3.1-Instruct effectively utilizes less than 30% of its claimed 128k context length for reasoning, exhibiting a pronounced performance gap between high- and low-resource languages and between retrieval and reasoning tasks, which off-the-shelf Retrieval Augmented Generation (RAG) alleviates to some extent without solving the long-context problem.
The experimental results in Tables 1, 5, 6 and Figures 5-8 consistently support the claimed gaps across languages, task types, and effective context length, as well as the partial mitigation by RAG.
“Our extensive experiments with an open-weight large language model (LLM) reveal a pronounced gap between high- and low-resource languages, particularly for tasks requiring the model to aggregate multiple facts or predict the absence of information. We also find that, in multilingual settings, LLMs effectively utilize less than 30% of their claimed context length. Although off-the-shelf Retrieval Augmented Generation helps alleviate this to a certain extent, it does not solve the long-context problem.” · Show in paper