By Biao Xiang, Soyeon Caren Han, Yihao DingSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2603-07931.git
Cite
Version link
https://openversions.com/index/arxiv-2603-07931/commit/d5589629292dc87c05de30e28b9bf74d98afb28c
Version identifier (commit hash)
d5589629292dc87c05de30e28b9bf74d98afb28c

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

BRIDGE is a benchmark of 11,857 multi-hop QA pairs over 262 long scientific papers that requires integrating evidence across text, tables, and figures, supports both chain-like and fan-out structures, and provides explicit multi-hop reasoning annotations for step-level evaluation.

In long multimodal document QA requiring multi-hop evidence, a Colpali-based RAG pipeline (retrieving top-k=3k=3 pages) substantially degrades end-to-end answer quality compared to Gemini-only baselines, with retrieval mismatch and evidence-missing as dominant failure modes.

Across unified evaluation on BRIDGE, ChatGPT performs best on audit_score (4.379–4.404), followed by Gemma (4.131–4.175), Gemini (3.728–3.989), and Qwen (3.489–3.605), while prompting strategy effects (Direct, CoT, Reflection) are model-dependent.

Lexical overlap metrics (ROUGE-L, BLEU) can diverge from LLM-as-a-judge correctness and grounding metrics, exhibiting strategy-induced stylistic drift even when judge metrics remain stable or improve.

Across question types, Causal-Reasoning is generally the most stable category for strong models (ChatGPT audit 4.766; Gemini audit 4.366), whereas Comparative questions are the hardest across all settings (ChatGPT drops to 3.691 audit and Colpali-RAG collapses to 1.002 audit).

Model performance degrades as the required evidence moves deeper into the document (from pages 1–2 to pages 11–20 and 21+).

Across all evaluated models, questions whose primary supporting evidence comes from tables are the most challenging, whereas figure-based evidence is relatively easier for stronger models.

Methods

Construct the BRIDGE corpus from 262 ArXiv research papers (2023–2025, primarily from top-tier NLP and computer vision venues such as ACL, EMNLP, CVPR, and ICCV), parsing each PDF with the Adobe PDF Extract API to obtain layout-aware text, tables, and figures with page indices and bounding-box metadata.

Source excerpts

“We construct the dataset from 262 research papers on ArXiv1, including PDF files and available LaTeX sources. To ensure structural quality, we focus on papers published at top-tier venues. The corpus spans recent computer science research (2023–2025), primarily in NLP and computer vision (e.g., ACL, EMNLP, CVPR, ICCV). Each paper is parsed using the Adobe PDF Extract API2 to extract text, tables, and figures with page indices and bounding-box metadata, enabling layout-aware document representations.” · Show in paper

Generate multi-hop QA pairs using a two-stage chain-of-thought prompting framework: Stage 1 (Structure Mining) extracts question-type-specific entity structures, and Stage 2 (Constraint-Guided Generation) conditions on these structures, formal question definitions, and multi-hop constraints to produce QA pairs with explicit evidence hops.

Source excerpts

“We adopt a two-stage prompting framework: Stage 1 (Structure Mining) extracts question-type-specific entity structures, and Stage 2 (Constraint-Guided Generation), conditions on these structures, formal question definitions, and multi-hop constraints to generate structured multi-hop QA pairs with explicit evidence hops.” · Show in paper

Filter generated QA pairs using a dual-stage pipeline consisting of (1) a rule-based pre-filter to remove malformed instances, unresolved references, and instances lacking numeric grounding or metadata, and (2) an LLM-as-a-judge framework using structured PDF anchors to flag hallucinations and single-hop shortcuts.

Source excerpts

“To ensure high data reliability, we employ a two-stage filtering pipeline: (1) a rule-based pre-filter that eliminates malformed instances, unresolved references, and instances lacking numeric grounding or metadata; and (2) an LLM-as-a-judge framework that evaluates grounding, faithfulness, and reasoning depth. By utilizing structured PDF anchors to flag hallucinations and "single-hop shortcuts," this process ensures a scalable, reproducible collection of high-confidence multi-hop QA pairs.” · Show in paper

Evaluate closed-source (Gemini-3, ChatGPT-5) and open-weight (Qwen3, Gemma-3) MLLMs under standardized input formatting and identical decoding settings (temperature = 0.2, max tokens = 2048), using ColPali as a page-level multimodal retriever (top-k=3k = 3 pages) for RAG experiments, and Qwen-Plus (temperature = 0.2) as an LLM-as-a-judge on a 0–5 scale alongside ROUGE and BLEU.

Source excerpts

“We evaluate representative MLLMs, including closed-source systems (Gemini-3 [5], ChatGPT-5[15]) and open-weight models (Qwen3 [1], Gemma-3 [4]), under a unified evaluation pipeline. All models are tested with standardized input formatting and identical decoding settings (temperature = 0.2, max tokens = 2048) to ensure fair comparison. For retrieval-augmented experiments, we employ ColPali as a multimodal retriever at the page level, retrieving top-k = 3 relevant pages per query. Retrieved content is concatenated with the prompt before generation. Then for result evaluation, we adopt Qwen-Plus (temperature = 0.2) as an LLM-as-a-judge to score final answer correctness and evidence alignment on a 0–5 scale. We further analyze reasoning behaviors using a structured error taxonomy. In addition, we report ROUGE [11] and BLEU [17] as complementary automatic metrics.” · Show in paper