Towards Automating Scientific Review with Google's Paper Assistant Tool
Source: arXiv:2606.28277 · Published 2026-06-26 · By Rajesh Jayaram, Drew Tyler, David Woodruff, Corinna Cortes, Yossi Matias, Vahab Mirrokni et al.
TL;DR
This paper addresses the growing bottleneck in scientific peer review arising from the rapid surge of AI-assisted scientific output, particularly in fields like mathematics, theoretical computer science, and machine learning. The traditional human review process struggles to keep pace with the volume and technical complexity of submissions, necessitating AI-augmented solutions for verification and review at scale. To framework this transition, the authors propose a taxonomy of four levels of AI-human collaboration in scientific evaluation, ranging from AI assisting authors to fully automated AI peer review.
In this context, they introduce the Paper Assistant Tool (PAT), an advanced agentic AI system designed for deep scientific review and verification. PAT segments manuscripts into semantically coherent parts, dynamically allocates compute resources based on section complexity, employs inference scaling with multiple passes of specialized deep review agents, and synthesizes the outputs with grounding techniques to reduce hallucinations. On the SPOT benchmark (a dataset of retracted papers with verified errors), PAT improves recall of mathematical errors by 34% over zero-shot Gemini 3.1 Pro baseline (achieving 89.7% recall). Pilot deployments at STOC and ICML prior to submission demonstrated PAT’s ability to catch critical errors that human reviewers had missed, leading to substantive improvements in papers and positive author feedback. The work highlights AI’s emerging role as a powerful tool in scientific reviewing while discussing societal and procedural challenges in moving toward greater automation.
Key findings
- PAT achieves 89.7% recall on mathematical and proof errors in the SPOT benchmark subset versus 55.2% for zero-shot Gemini 3.1 Pro and 21.1% original SPOT SOTA.
- Inference scaling in PAT yields a 34% absolute gain in error recall over single zero-shot model calls.
- In pilot use at STOC 2026 (n=124 authors) and ICML 2026 (n=733 authors), over 90% of authors found PAT feedback very or mostly helpful, with over 90% wanting to use PAT again.
- PAT identified significant theory errors requiring >1 hour of fix time in 11.6% of STOC authors and 35.4% of ICML authors.
- 31% of ICML authors ran new experiments after receiving PAT reviews, reflecting substantive impact on empirical validation.
- PAT uncovers subtle bugs and mathematically significant typos missed by human reviewers (e.g., off-by-one errors, contradiction in proofs).
- Hallucination and PDF parsing issues remain failure modes; over half the authors considered feedback mostly or fully grounded.
- The volume of submissions to AI conferences has exploded, with 62.9% YOY growth estimated for 2026, demonstrating acute scaling pressure on peer review.
Threat model
The adversary is an untrusted or fallible author submitting manuscripts that may contain errors, either accidental or from AI assistance. The system assumes access to full manuscripts and aims to detect factual, mathematical, and logical errors. It does not explicitly defend against authors deliberately adversarially crafting inputs to evade detection or generate misleading outputs. Human reviewers retain ultimate authority and responsibility.
Methodology — deep read
Threat Model & Assumptions: The adversary is not explicitly defined as malicious but the system is designed to catch natural human and AI-generated scientific errors in manuscripts. It assumes full access to the paper but does not address adversarial attempts to fool the system. Human reviewers retain final responsibility.
Data: The main evaluation data is the SPOT benchmark (Son et al. 2025), consisting of papers with verified technical errors and retractions. For quantitative evaluation, a subset of 26 papers with 29 equation/proof errors in Mathematics and Computer Science categories was used. The dataset includes ground truth error annotations validated by humans and a specialized LLM-based logical equivalence grader was used to assess detection accuracy.
Architecture / Algorithm: PAT consists of four stages: (1) Segmenting manuscripts into logical thematic sections (introduction, theory, methodology, experiments, etc.), allowing overlapping and non-contiguous segmentation; (2) Adaptive compute budgeting that allocates more inference effort to complex segments and less to simpler ones; (3) Deep Review Agents powered by advanced versions of Google's Gemini Deep Think models running multiple coordinated model calls per segment to deeply verify claims, proofs, and experiments; (4) A Synthesis Agent combines segment-level reviews, performs grounding checks using Google Search to counter hallucinations, deduplicates issues, and outputs a comprehensive evaluation report.
This inference scaling pipeline improves precision and recall over naive single or Pass@k model calls by orchestrating the compute allocation and integrating external grounding.
Training Regime: No explicit training or fine-tuning is described; PAT uses inference scaling and specialized agents built on pre-existing foundation models (Gemini 2.5 and 3.1 Pro). Hyperparameters governing segment complexity scoring and compute allocation are selected based on paper properties but details are not fully enumerated.
Evaluation Protocol: Error detection accuracy was measured by comparing generated reviews against ground truth in the SPOT benchmark subset using a logical equivalence-aware automated grader audited manually. Baselines included original SPOT reported results and zero-shot Gemini 3.1 Pro calls without inference scaling. Ablations with and without scaling steps are discussed implicitly through baseline comparisons. Pilot deployments at STOC and ICML conferences gathered qualitative and quantitative author feedback through surveys measuring perceived helpfulness, grounding, error detection, and downstream paper improvements.
Reproducibility: The underlying foundation models (Gemini) and PAT framework are proprietary to Google and currently not publicly released. The SPOT benchmark is publicly available. Exact hyperparameters, seeds, and code for PAT are not provided. Evaluation details such as human auditing of autograder outputs increase reliability of reported metrics.
Technical innovations
- Inference scaling pipeline with dynamic compute budgeting allocating focused compute to complex manuscript segments to improve deep verification.
- Multi-agent architecture combining segmenter agent, specialized deep review agents, and a synthesis agent that grounds outputs through web search.
- Use of logical equivalence-aware grading rather than exact keyword matching to evaluate error detection accuracy on scientific claims.
- Pilot integration of AI reviewing tool as a pre-submission author aid in major top-tier computer science conferences to measure impact.
Datasets
- SPOT Benchmark (Math/CS subset) — 26 papers with 29 equation/proof errors — public open dataset with verified retractions/errors
Baselines vs proposed
- Original SPOT SOTA: Detection Accuracy = 21.1% vs PAT: 89.7%
- Gemini 3.1 Pro zero-shot: Detection Accuracy = 55.2% vs PAT: 89.7%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28277.
Fig 1 (page 1).
Limitations
- Hallucinations remain a failure mode, requiring human verification and grounding to avoid false claims.
- PDF parsing errors and formatting issues cause occasional failures or overlooked content.
- Feedback quality varies, with roughly half the authors rating feedback as mostly or fully grounded, implying room for improvement.
- PAT currently focuses mainly on objective error detection; it does not produce subjective paper rankings or acceptance recommendations.
- No explicit adversarial robustness evaluation or testing under distribution shift (e.g. other scientific domains) is reported.
- Code, model weights, and exact configurations of PAT are proprietary and not publicly available, limiting reproducibility.
Open questions / follow-ons
- How can AI reviewing tools handle broader scientific domains with diverse writing and verification conventions beyond Math/CS?
- What are best practices and policy frameworks to govern ethical AI use in peer review to mitigate hallucination risks and maintain reviewer accountability?
- How to extend PAT-style inference scaling approaches to subjective assessments and paper evaluation metrics beyond error detection?
- What methods can ensure robustness and fairness in automated review systems, reducing biases and vulnerabilities to adversarial gaming?
Why it matters for bot defense
For practitioners in bot defense and CAPTCHA design, this work is illustrative of how inference scaling and multi-agent orchestration can enhance automated detection of subtle, context-dependent errors in highly technical texts—a challenge analogous to detecting sophisticated bot interactions that evade single-model scrutiny. The segmentation and focused compute allocation strategies can inspire analogous approaches in analyzing complex user behaviors or multi-step bot tactics.
Furthermore, the deployment of PAT as an assistive tool that preserves human oversight aligns with principles in bot defense emphasizing collaboration between AI detection and human adjudication to balance precision and recall. The taxonomy of AI roles in review also highlights the importance of clear operational boundaries and feedback mechanisms when integrating AI agents into high-stakes evaluation workflows, which is crucial for responsible CAPTCHA systems that avoid excessive user friction or false positives.
Cite
@article{arxiv2606_28277,
title={ Towards Automating Scientific Review with Google's Paper Assistant Tool },
author={ Rajesh Jayaram and Drew Tyler and David Woodruff and Corinna Cortes and Yossi Matias and Vahab Mirrokni and Vincent Cohen-Addad },
journal={arXiv preprint arXiv:2606.28277},
year={ 2026 },
url={https://arxiv.org/abs/2606.28277}
}