By Amey Hengle, Prasoon Bajpai, Soham Dan, Tanmoy ChakrabortySourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2504-12845.git
Cite
Version link
https://openversions.com/index/arxiv-2504-12845/commit/340ec1f214173e780fcc4ad1b183cebd12c93076
Version identifier (commit hash)
340ec1f214173e780fcc4ad1b183cebd12c93076

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2504.12845v1 OpenVersions: 337ae0f6-8455-4d01-92cb-df3395dcba12

Posted

Claims

Findings

Headline finding

In multilingual settings, Llama-3.1-Instruct effectively utilizes less than 30% of its claimed 128k context length for reasoning, exhibiting a pronounced performance gap between high- and low-resource languages and between retrieval and reasoning tasks, which off-the-shelf Retrieval Augmented Generation (RAG) alleviates to some extent without solving the long-context problem.

Llama-3.1-Instruct effectively utilizes only up to 25% to 30% of its advertised 128k context length for multilingual reasoning tasks, with an effective context window (ECW) of 16k tokens for the best prompt-only methods and 32k tokens for the best RAG methods.

Multilingual reasoning performance of Llama-3.1-Instruct drops sharply as linguistic distance from English increases, in both short- and long-context settings, with English achieving the highest performance and Chinese the lowest.

While Llama-3.1-Instruct is effective at retrieving isolated facts across languages and context lengths, its performance degrades substantially on tasks requiring multi-hop inference, aggregation, or epistemic reasoning (uncertainty).

Off-the-shelf RAG methods (JinaAI and MpNet) improve stability and show a smaller accuracy drop across increasing context lengths than prompt-only methods, but their accuracy at longer context windows remains approximately 20% to 40% below the best short-context baseline.

Prompt-only methods (zeroshot, in-context translation, CoT, and fewshot) lose more than 50% of their baseline accuracy on average at longer context windows (>= 32k tokens), with nearly all deteriorating sharply after 8k to 16k tokens.

Synthetic distractors generated by GPT-4 confuse the model slightly more than natural distractors from MLQA or random noise.

The sentence-level machine translation process used to construct MLRBench from English bAbI preserves semantic alignment across all six target languages, with average information loss below 2% on back-translation and no misleading translations found in a 5% manual review.

Methods

MLRBench is constructed by adapting 1,000 English bAbI QA instances across seven reasoning tasks into seven languages (en, de, es, hi, ar, vi, zh), replacing character names with culturally appropriate alternatives and translating independent sentences via the Google Translate API.

Source excerpts

“To make the dataset more representative across languages, we replace character names in bAbI with culturally appropriate alternatives for each target language, as shown in Table A.1. Following this pre-processing step, we use the Google Translate API to translate the passage, question, and answer from English to all selected languages.” · Show in paper

Long-context prompts P are constructed by interleaving chronologically ordered sentences of an input passage IP randomly within k semantically similar distractor passages D retrieved via cosine similarity, supporting GPT-4 synthetic distractors, MLQA natural distractors, and random noise.

Source excerpts

“Specifically, given a task instance (passage, query) and a corpus of background texts, we retrieve the top-k most semantically similar distractor passages based on cosine similarity. The value of k is adjusted dynamically to control the overall length of the prompt, allowing our setup to scale the long contexts to any arbitrary length as needed.” · Show in paper

“The long context P is constructed by placing sentences in IP randomly within the distractor passages D. This is done while maintaining the original order of sentences in IP.” · Show in paper

Experiments evaluate meta-llama/Llama-3.1-8B-Instruct across context lengths from 0k to 128k tokens using four prompting strategies (zeroshot, fewshot, chain-of-thoughts, and in-context translation) and two off-the-shelf RAG retrievers (Jina-reranker V2 Base Multilingual and paraphrase-multilingual-mpnet-base-v2) with top-k sentence retrieval.

Source excerpts

“We conduct all our experiments using the open-weight Llama 3.1-Instruct model, which is an instruction-finetuned version of Llama 3.1 (Meta AI 2024). ... For our RAG experiments, we make use of an off-the-shelf and a specialised multilingual models – Jina-reranker (JinaAI) (Multilingual 2024) and paraphrase-multilingual-mpnet-base-v2 (MPNet) (Reimers and Gurevych 2020), respectively.” · Show in paper

Model performance is evaluated using exact accuracy and Recall@k, with binomial significance tests conducted across evaluation sample sizes from 10 to 500 using standard error SE=p(1−p)sSE = \sqrt{\frac{p(1-p)}{s}}.

Source excerpts

“We use exact accuracy as our primary evaluation metric. ... Additionally, we evaluate RAG performance using Recall@k, which measures the proportion of relevant sentences retrieved for a given question” · Show in paper

“Let Binomial(1, p) represent the binomial distribution, where p is the probability of success on an individual trial. The standard error (SE) is computed using the formula: SE = sqrt(p(1-p)/s)” · Show in paper