Contributors @natalie, @tjreed-research, @tjreed15SourceGitHub
Fork
Clone
git clone https://openversions.com/git/natalie/archive-57247ede1f9d.git
Cite
Version link
https://openversions.com/natalie/archive-57247ede1f9d/commit/74a852c8ec55cebb6c353961fd8ac1d5508d968f
Version identifier (commit hash)
74a852c8ec55cebb6c353961fd8ac1d5508d968f

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Across disciplines, AI reviews are more structured and engage more directly with a paper's major claims than human reviews, while human judgment remains essential for evaluating contribution and shaping editorial decisions, supporting a complementary model rather than AI replacement.

AI reviews are more structured than human reviews across all three disciplines (ICLR 2025, Nature Human Behaviour, and eLife), with R3 leading on justification (96.4–97.3%96.4\text{--}97.3\%, all padj<.001p_{\text{adj}} < .001, ∣d∣=1.74|d| = 1.74 vs. human) and actionability (99.2–99.9%99.2\text{--}99.9\%, all padj<.001p_{\text{adj}} < .001, ∣d∣=1.38|d| = 1.38 vs. human).

R3 (63.5–79.0%63.5\text{--}79.0\%) and Gemini 3 Pro (66.3–78.0%66.3\text{--}78.0\%) achieve higher claim-mapping rates than all other sources across all three venues (all padj<.001p_{\text{adj}} < .001, ∣d∣≥0.48|d| \ge 0.48), suggesting they engage more directly with major claims.

Across all venues, human reviews have the most balanced stance profile, with the highest supportive (11.1–26.8%11.1\text{--}26.8\%) and neutral (8.0–8.2%8.0\text{--}8.2\%) rates and the lowest critical rates (65.1–80.9%65.1\text{--}80.9\%) in most venues, whereas R3 is the most critical of all sources (98.0–98.3%98.0\text{--}98.3\%).

R3 achieves the highest consequential rate across all three venues (87.9–93.2%87.9\text{--}93.2\%), compared to 65.5–77.5%65.5\text{--}77.5\% for Gemini 3 Pro (all padj<.001p_{\text{adj}} < .001, ∣d∣≥0.79|d| \ge 0.79), 73.7–81.4%73.7\text{--}81.4\% for GPT-5.2 (all padj<.001p_{\text{adj}} < .001, ∣d∣≥0.50|d| \ge 0.50), and 60.0–67.7%60.0\text{--}67.7\% for humans (all padj<.001p_{\text{adj}} < .001, ∣d∣≥1.00|d| \ge 1.00).

When restricting to critical comments, human reviewers rank first on consequential rate on more individual papers than any AI source despite a lower average, and individual human reviewer consequential rates follow a bimodal distribution with peaks near 0%0\% and 100%100\% across all three venues.

Human and AI sources differ systematically in critique focus: human reviewers have the highest fraction of comments related to contribution across all venues (17.5–25.2%17.5\text{--}25.2\%) and clarity in ICLR (23.8%23.8\%) and Nature Human Behaviour (28.8%28.8\%), whereas AI sources devote more than half of their attention to validity and sufficiency across all venues, with R3 leading in validity (34.8–46.9%34.8\text{--}46.9\%).

The Gemini evaluator used throughout the study shows no self-favoritism on any assessment metric, and Gemini 3 Pro and GPT-5.2 evaluators agree on the first-place source for all metrics and on exact rank ordering for all but one metric.

Methods

ReviewBench is implemented as a four-stage pipeline consisting of raw data fetchers, a canonicalization layer into papers and review comments tables, AI review generators, and a processor for claim extraction, comment assessment, and scoring.

Source excerpts

“ReviewBench is a publicly available [24] four-stage pipeline consisting of raw data fetchers, a layer to canonicalize raw data into papers and review comments tables, AI review generators, and a processor to extract claims from papers, assess comments, and score.” · Show in paper

Papers and human peer reviews were collected across three venues without topic or acceptance filtering: ICLR 2025 (n = 1,000 via OpenReview API v2), Nature Human Behaviour (n = 142 after excluding 17 lacking structured reviewer remarks), and eLife (n = 1,000 from the OpenEvalProject evals repository).

Source excerpts

“ICLR 2025 submissions and reviews (n = 1,000) were retrieved via the OpenReview API v2. Nature Human Behaviour Open Access research articles with linked peer review files (n = 159) were scraped from the journal website. 17 articles were excluded whose peer review files did not contain structured reviewer remarks. eLife papers and peer reviews (n = 1,000) were loaded from a locally cloned copy of the OpenEvalProject evals repository [22] containing manuscript text and peer review text.” · Show in paper

Venue-specific parsers canonicalized paper metadata and split reviewer letters into individual comments (discarding comments shorter than 20 characters), followed by one-time repair scripts for splitting artifacts.

Source excerpts

“For each venue, a venue-specific parser extracted structured paper metadata into a canonical papers table. Reviewer letters were split into individual comments and stored in a comments table. The splitting logic detects whether the text uses numbered lists, bullet lists, or paragraphs, and splits accordingly, discarding comments shorter than 20 characters.” · Show in paper

AI reviews were generated using Reviewer3.com (R3, multi-agent architecture in LLMcfg01.06.2026) and single-model baselines Gemini 3 Pro (gemini-3-pro-preview) and GPT-5.2 (gpt-5.2) at temperature 0 with a minimal uniform prompt.

Source excerpts

“R3 reviews were generated for all three venues using a multi-agent architecture with specialist reviewer agents in LLM configuration LLMcfg01.06.2026. R3 accepts either a PDF or manuscript text as input and requires no venue-specific modification. Frontier LLM reviews were generated with Gemini 3 Pro (gemini-3-pro-preview) and GPT-5.2 (gpt-5.2) to establish a baseline for single-model LLM capability.” · Show in paper

For each paper, 3–7 major claims and verbatim source excerpts were extracted using gemini-3-pro-preview at temperature 0.

Source excerpts

“For each paper, the manuscript PDF (ICLR, Nature Human Behaviour) or text (eLife) is supplied to gemini-3-pro-preview at temperature 0 with a prompt instructing it to identify 3–7 major claims.” · Show in paper

Every comment from all four sources was assessed using three separate LLM calls to gemini-3-pro-preview at temperature 0 with a cached manuscript context: comment parsing (specification, justification, suggested remedy, anchor), claim mapping (claim ID, stance, consequential label), and critique typing (validity, sufficiency, contribution, clarity, transparency).

Source excerpts

“For each paper, the pipeline assesses every comment from all four sources (Human, R3, Gemini 3 Pro, GPT-5.2) using three separate LLM calls to gemini-3-pro-preview at temperature 0, each with a dedicated prompt.” · Show in paper

Twelve per-paper metrics were computed in Python, and pairwise source comparisons within each venue were conducted using two-tailed Wilcoxon signed-rank tests with Holm–Bonferroni correction and Cohen's d effect sizes.

Source excerpts

“We therefore use the Wilcoxon signed-rank test for all pairwise comparisons, with the paper as the unit of analysis, because comments within a paper are not independent. Within each venue, all four sources are assessed on the same papers. Raw p-values are Holm–Bonferroni corrected jointly. Because the large sample gives high power to detect trivially small differences, we report Cohen’s d as a measure of practical effect size.” · Show in paper

A cross-model evaluator study was conducted on 50 randomly sampled papers by re-assessing each source's comments with GPT-5.2 using the same prompts, claims, and post-processing pipeline.

Source excerpts

“To assess whether the evaluating LLM introduces self-preference bias, we randomly sampled 50 papers and assessed each source’s comments using GPT-5.2 with the same prompt, claims, and post-processing pipeline used by the primary Gemini evaluator.” · Show in paper

Research files

paper.pdf

Open the manuscript PDF