PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image

Source: arXiv:2607.19261 · Published 2026-07-21 · By Dankai Liao, Tianyi Zhang, Yufeng Wu, Xinyue Zhang, Qiaochu Xue, Zeyu Liu et al.

TL;DR

PathAgentBench addresses a critical gap in computational pathology: the evaluation of vision-language models (VLMs) not only on reasoning over diagnostic evidence but also on autonomous evidence acquisition directly from gigapixel whole-slide images (WSIs). Existing pathology benchmarks focus primarily on interpreting preselected patches or slide features, ignoring the essential task of actively seeking diagnostically relevant regions across scales. This work introduces PathAgentBench, the first large-scale benchmark designed as a hierarchical diagnostic tree spanning multiple magnifications (2.5×, 10×, 40×) and linking nested image regions to pathological findings and overall slide diagnoses.

PathAgentBench operationalizes four complementary tasks: image-to-text matching for evidence interpretation, text-to-image retrieval for evidence verification, diagnostic-region localization for evidence acquisition, and multi-scale diagnostic reasoning for evidence integration. It includes 1,822 TCGA WSIs with 17,135 diagnostic paths annotated by 10 pathologists, plus a private 190-WSI breast cancer cohort for autonomous exploration evaluation. Evaluating 20 general-purpose, medical, and pathology-specialized models reveals a stark divide: models achieve over 93% accuracy in multi-scale reasoning and >50% in cross-modal matching but struggle significantly on evidence acquisition, with text-guided localization mean IoU below 0.09 and autonomous exploration hit rates sharply declining across magnifications. These results highlight that evidence acquisition—actively navigating and localizing diagnostic regions on WSIs—is the principal bottleneck for pathology agents. PathAgentBench offers a unified framework to measure and improve all stages of the evidence-seeking diagnostic process.

Key findings

• Leading open-weight models achieve over 93% accuracy on multi-scale diagnostic reasoning task (Task 4).
• Top models achieve over 50% accuracy on cross-modal image-to-text (Task 1) and text-to-image retrieval (Task 2) tasks.
• Diagnostic-region localization (Task 3) remains extremely challenging: the best text-guided localization mean intersection-over-union (mIoU) is below 0.09, worse than a simple parent-box-center heuristic with mIoU up to ~0.28.
• During autonomous whole-slide exploration (Task 3 Mode B), unconditional hit rate drops sharply from 0.522 at 2.5× magnification to 0.185 at 10× and 0.020 at 40×, evidencing severe end-to-end attrition.
• Pathology-specialized VLMs performed poorly on patch-level recognition tasks, often below random choice baseline (~25%).
• Expanding exploration budget modestly improves but does not resolve localization performance bottlenecks (e.g., hit rate at 40× increases from 2.06% to 6.25% with larger branching).
• Multi-scale multi-magnification inputs improve diagnostic reasoning accuracy for 15 of 16 models compared to any single magnification input.
• Closed-source large VLMs outperform open models on localization metrics but still exhibit very low IoU scores and rapid drop-off in exploration coverage.

Threat model

The adversary in this context is the computational model tasked with autonomously navigating whole-slide pathological images to seek evidence without prior region proposals or external guidance. It must localize, verify, interpret, and integrate multi-scale features to produce diagnoses. The adversary does not have access to curated evidence but must choose where and what to examine. It cannot bypass the hierarchical slide structure or access privileged annotations during inference.

Methodology — deep read

1. Threat model & assumptions: The adversary is essentially conceptualized as a challenge for models to autonomously seek, verify, and interpret evidence on gigantic WSIs. The models do not have pre-selected evidence but must navigate and acquire diagnostically informative regions from scratch. The adversary can only work within the constraints of vision-language models and multi-scale image pyramids.

2. Data provenance and annotation: The benchmark includes 1,822 WSIs from TCGA annotated by 10 board-certified pathologists using a diagnostic-tree framework. Each WSI's diagnostic tree nests 2.5×, 10×, and 40× magnification regions, with bounding boxes and pathologist-authored findings per node, plus diagnostic route summaries. The annotation pipeline involved primary annotators selecting paths, findings, and diagnoses, reviewed by secondary experts, iteratively until quality standards were met. Additionally, a private 190 breast cancer WSI cohort with tumor polygons is used for autonomous navigation evaluation.

3. Architecture and task formulation: PathAgentBench decomposes WSI diagnosis into four core tasks:

   • T1: Evidence interpretation (image-to-text matching) asks if the model can select the correct pathologist-authored description among 4 options from a given image patch at a specific magnification and organ.
   • T2: Evidence verification (text-to-image retrieval) asks if the model can select the correct image patch among 4 candidates for a given textual finding.
   • T3: Evidence acquisition - evaluated in two modes: Mode A (text-guided localization): given a WSI and text, locate the corresponding diagnostic region bounding box at specified magnification, evaluated by mean IoU and fraction of boxes with IoU≥0.3. Mode B (autonomous exploration): the agent scores tiles at increasing magnifications and selects top-K to traverse hierarchically, evaluated by conditional and unconditional hit rates referencing ground-truth tumor regions.
   • T4: Evidence integration (multi-scale diagnostic reasoning) asks the model to integrate findings from the three magnifications along a diagnostic path and select the correct diagnosis (from 4 choices).

The tasks are carefully designed to isolate components:

• T1, T2, and T4 fix evidence and test reasoning,
• T3 evaluates the policy for evidence acquisition and navigation.

4. Training regime: The evaluated models vary in size (4B to >1T params), training domain (general-purpose, medical, pathology-specialized), with open and closed weights. Models are off-the-shelf checkpoints from prior work. No further fine-tuning for PathAgentBench is reported.

5. Evaluation protocol: T1/T2 use thousands of 4-option multiple choice questions with carefully mined distractors using a cosine similarity-based clustering (MiniLM embeddings). Performance is exact-match accuracy.

T3 Mode A tests localization on a 50-slide subset with 3 queries per slide, constrained to parent-box hierarchies, evaluated by mean IoU and recall at IoU≥0.3.

T3 Mode B evaluates autonomous tile scoring/navigation on the 190-slide breast cohort with retention schedules (6,3,2) tiles per level. Metrics are conditional and unconditional hit rates measuring fraction of true positive tumor tiles retained during hierarchical selection.

T4 uses over 51k 4-option multiple choice questions for diagnostic reasoning (triage, diagnose, refine questions) with exact-match accuracy.

Baselines include random choice, frequent choice, grid centroid, tissue density, parent box centers, and non-VLM retrieval systems.

6. Reproducibility: The authors released the PathAgentBench benchmark and evaluation framework on GitHub, including the TCGA diagnostic trees and code for task evaluation. Model checkpoints are public or analyst estimated. The private breast cohort is not public.

Example end-to-end evaluation for localization (Task 3 Mode A): Given a textual target such as “High-density malignant epithelioid cell population” at 10× magnification, the model issues a sequence of calls to extract regions of interest, then returns a bounding box prediction constrained in size to the ground truth. The IoU between predicted and annotated boxes is computed, averaged over 150 queries in the 50-slide subset. Across models, mean IoU remained low (<0.09), and larger exploration budgets degraded results, revealing difficulty in precise localization on WSIs with text guidance alone.

Technical innovations

• New multi-scale diagnostic tree framework explicitly modeling evidence acquisition and reasoning on hierarchical nested regions in WSIs.
• Four complementary tasks isolating core stages of pathology VLM workflow: interpretation, verification, acquisition, and integration.
• Large-scale expert-annotated dataset of 1,822 WSIs with 17,135 multi-magnification diagnostic paths providing dense pathologist-authored findings and diagnoses.
• Evaluation of autonomous whole-slide navigation via hierarchical tile scoring contrasting text-guided localization and unguided exploration modes.

Datasets

• TCGA diagnostic trees — 1,822 WSIs, 17,135 paths, 10 pathologists annotated — public TCGA slides with in-house annotations
• Private breast cancer cohort — 190 WSIs with tumor polygons — private (not public)

Baselines vs proposed

• Random choice: ~25% accuracy on interpretation and verification tasks vs top model Gemini-3-Flash: 63.5% (T1) / 67.7% (T2)
• Parent-box-center heuristic: mean IoU up to 0.28 vs best VLM localization IoU < 0.09 (Task 3A)
• Random sampling baseline in autonomous exploration: unconditional hit rates ~30.7% at 2.5× vs Gemini model 52.2% (Task 3B)
• Gemini-3-Flash multi-scale diagnostic reasoning accuracy 91.2% vs pathology-specialized Patho-R1 70.1% (Task 4)
• Patch-level pathological recognition: pathology-specialized Quilt-LLaVA 7.0% (below random 25%) vs general VLMs over 50% (T1)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.19261.

Fig 1: Motivation for PathAgentBench. Current pathology

Fig 2: Diagnostic-tree formulation of evidence seeking. A WSI is the root; at each magnification, selected regions become

Fig 3: Overview of PathAgentBench. (a) Four tasks and their evaluation axes; numbers denote task-specific evaluation budgets.

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Private breast cancer cohort used for autonomous exploration evaluation is not public, limiting reproducibility in Mode B.
• Localization task (T3) evaluation limited to small subsets due to annotation costs; broader evaluation on all WSIs would be helpful.
• Models evaluated off-the-shelf without fine-tuning on this benchmark; possible performance gains from dedicated training remain unexplored.
• No adversarial robustness testing or evaluation of models under domain shift.
• Marker and subtype prediction (Refine questions) remain challenging with lower accuracy but detailed error analysis is limited.
• Diagnostic tree formulation may not capture all real-world clinical workflow nuances across specialties.

Open questions / follow-ons

• How can evidence acquisition policies be improved beyond hierarchical tile scoring to reduce early pruning and better localize regions at higher magnifications?
• What specialized training or adaptation methods can close the gap between reasoning over curated evidence and autonomous evidence acquisition?
• Can multi-agent or reinforcement learning approaches better model complex diagnostic navigation on gigapixel WSIs?
• How generalizable are current VLMs and the PathAgentBench diagnostic-tree schema across pathology domains beyond the organs and diagnoses represented here?

Why it matters for bot defense

PathAgentBench’s framework emphasizes the critical distinction between passive interpretation of supplied image regions and active, autonomous evidence acquisition from ultra-high-resolution images. For bot-defense and CAPTCHA practitioners, this highlights the importance of evaluating AI systems not only on their ability to reason over fixed inputs but also on their capability to intelligently navigate large, complex spatial domains under multi-step instructions. The staged tasks and hierarchical diagnostic tree model could inspire similar multi-stage challenge designs that test agentic AI behavior in bot detection scenarios.

Moreover, the severe performance degradation in localization and exploration despite strong reasoning capabilities suggests that AI agents can excel at high-level understanding once evidence is acquired but may fail at strategic information gathering. This insight can guide bot-defense engineers to develop evaluation benchmarks and challenge heuristics that probe not just final inference accuracy but also the underlying search and acquisition logic of intelligent agents. The PathAgentBench concept may serve as a template for constructing layered agentic benchmarks in other multimodal domains beyond pathology.

Cite

@article{arxiv2607_19261,
  title={ PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image },
  author={ Dankai Liao and Tianyi Zhang and Yufeng Wu and Xinyue Zhang and Qiaochu Xue and Zeyu Liu and Dachun Zhao and Linghan Cai and Yueming Jin },
  journal={arXiv preprint arXiv:2607.19261},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.19261}
}

Read the full paper

• arXiv:2607.19261 (abstract)
• arXiv:2607.19261 (PDF)