Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology

Source: arXiv:2606.20477 · Published 2026-06-18 · By Yusuf Salcan, Simon Ging, Robin Schirrmeister, Philipp Arnold, Elmar Kotter, Behzad Bozorgtabar et al.

TL;DR

This paper addresses a major challenge in medical vision-language models (VLMs) for radiology: enabling reliable spatial grounding of model outputs without relying on costly manual annotations. The authors contribute RefRad2D, a large bilingual (German/English) dataset of 1.2 million clinical 2D CT and MRI image-text pairs, from a decade of clinical data, enriched with automated spatial grounding labels via large language model (LLM) curation and automated anatomical segmentation. On this data, they train RadGrounder, a multi-task VLM that jointly performs medical report generation, visual question answering (VQA), and spatial grounding through bounding-box detection or segmentation. RadGrounder matches or surpasses expert medical VLMs on established external VQA benchmarks (Slake and VQA-RAD), showing good transferability from the large clinical dataset.

A key finding is that adding spatial grounding supervision does not degrade the linguistic quality of report generation or VQA, allowing for spatially verifiable outputs at no performance cost. They also demonstrate that the proposed token-based bounding-box detection approach provides effective localization without needing expensive auxiliary segmentation heads or extra losses. Extensive experiments explore training configurations, anchoring the system architecture on the PaliGemma 2 VLM framework. This work significantly scales 2D slice-level radiology VLM training, addressing prior data scarcity and grounding limitations, advancing towards clinically trustworthy spatially grounded medical VLMs.

Key findings

• RefRad2D dataset contains 1.2 million bilingual (German/English) CT and MRI 2D image-text pairs with 236,157 automatically generated spatially grounded slice-text pairs.
• RadGrounder achieves competitive VQA performance on Slake (F1=87.7, Closed Accuracy=90.3) and VQA-RAD (Open F1=50.7), matching or surpassing existing medical VLMs like BiomedGPT-B and Med-Gemini.
• Adding RefRad2D clinical data as pretraining improves open-ended VQA F1 on Slake from 86.9 to 87.7 and on VQA-RAD from 47.9 to 50.7 versus fine-tuning on downstream datasets alone.
• Token-based bounding box detection yields higher G-IoU (43.6) for localization than segmentation (36.9) while requiring no auxiliary decoder or additional loss terms.
• Freezing the SigLIP vision encoder pretrained on RefRad2D achieves comparable results to unfreezing it, reducing memory and training time.
• Spatial grounding supervision does not degrade report generation quality; CIDEr scores reach ~39.2 on internal test with LLMScore indicating semantic correctness.
• Novel LLMScore metric, using a 27B parameter LLM as judge, correlates strongly (Pearson r = 0.977) to expert radiologists’ factuality scores on VQA-RAD test samples.
• Domain mixing experiments show internal-only training achieves best report scores but poor external VQA generalization, whereas mixing internal and external data improves generalization.

Threat model

n/a - The paper does not focus on attacker modeling or adversarial threats but rather on robust training and evaluation of spatially grounded vision-language models in radiology with automated annotations.

Methodology — deep read

1. Threat Model & Assumptions: The work targets vision-language models for radiology image slices (CT and MRI) aiming to produce text outputs spatially grounded in image regions. There is no direct adversarial threat modeling, but challenges include avoiding hallucinations and ensuring verifiable spatial localization without manual annotations.

2. Data: RefRad2D is derived from a decade of clinical data from a single university hospital, with 1.2M CT (945k) and MRI (321k) 2D slice-caption pairs. Text is bilingual (German original and English translations) with translations refined by large LLMs for quality and bias reduction. Spatial grounding for 236,157 slices uses 3D anatomical masks from TotalSegmentator (121 unified classes) projected to 2D slices, linked to text via an LLM-extracted class matching procedure. A synthetic VQA dataset with ~9.6M QA pairs was generated via LLM prompting from clinical findings and metadata.

3. Architecture: RadGrounder is built on the PaliGemma 2 framework, combining a SigLIP-So400m vision encoder with the Gemma-2B language decoder. The model consumes image patches converted to visual tokens concatenated with text tokens autoregressively. Spatial grounding is implemented in two variants: (a) token-based bounding-box detection via extending the text vocabulary to include discretized coordinate tokens and class identifiers, trained end-to-end with autoregressive cross-entropy loss; (b) an auxiliary segmentation head based on a lightweight mask decoder following Two-Way Transformer design, trained with additional focal and Dice losses.

4. Training Regime: Pretraining includes contrastive adaptation of the SigLIP encoder on RefRad2D before multi-task training. Multi-task training jointly optimizes report generation, VQA, and grounding objectives, alternating batches. Models were trained on 1 NVIDIA H100 GPU for 6 epochs, batch size 24 with gradient accumulation, using Adafactor optimizer (lr=5e-5). Data augmentations include random scaling, cropping, and intensity shifts.

5. Evaluation Protocol: Internal test sets contain 2,000 samples per split for report, VQA, and grounding tasks. External validation uses Slake and VQA-RAD benchmarks with their standardized splits. Metrics include CIDEr and a novel LLMScore for text generation (an LLM-based factuality judge correlating with radiologists), F1 and accuracy for VQA, and Grounding-IoU (G-IoU), which aligns predicted and ground-truth anatomical entities spatially and semantically via embedding similarity weighted IoU. Comparisons were made against recent medical VLM baselines (PaliGemma 2, Gemma 3, MedGemma, BiomedGPT-B, LLaVA-Med, Med-Gemini).

6. Reproducibility: Code and pretrained RadGrounder models are promised upon acceptance. RefRad2D is derived from a clinical dataset not publicly released. The paper describes dataset construction pipelines and training details with hyperparameters. Evaluation scripts and the LLMScore metric details are explained but not explicitly open-sourced here.

Example: For a chest CT slice with a nodular lesion, RadGrounder predicts a bounding box that spatially overlaps the ground truth with a G-IoU of 0.9497 and a high semantic match. The textual report generation achieves a CIDEr of 0.1373 but a high LLMScore of 0.75 indicating semantic accuracy despite lexical variation. This demonstrates the model’s ability to ground findings and generate clinically relevant reports jointly.

Technical innovations

• RefRad2D - a scalable bilingual dataset of 1.2M clinical 2D CT and MRI image-text pairs with automated spatial grounding annotations derived from 3D anatomical segmentation and LLM-based text curation.
• RadGrounder - a multi-task VLM architecture that jointly performs medical report generation, visual question answering, and spatial grounding via token-based bounding box detection without auxiliary segmentation.
• Token-based bounding box detection as a novel grounding supervision performed end-to-end as a text generation task by extending the vocabulary with discretized coordinate tokens and class identifiers.
• Development of LLMScore: an LLM-based evaluation metric for medical report factuality and semantic correctness, correlating strongly (r=0.977) with radiologist ratings.
• Grounding-IoU (G-IoU), a metric that combines spatial Intersection over Union with semantic similarity of anatomical tokens for grounded localization evaluation.

Datasets

• RefRad2D — 1.2 million image-text pairs — clinical CT/MRI data from a single university hospital (non-public)
• RefRad2D-Grounded — 236,157 slice-text pairs with spatial grounding masks/bounding boxes — derived automatically from RefRad2D
• RefRad2D-VQA — ~9.6 million QA pairs synthetically generated from clinical captions — derived automatically from RefRad2D
• Slake — 11,919 total QA pairs (9,849 training / 2,070 testing) — public medical VQA benchmark
• VQA-RAD — 3,515 total QA pairs (3,064 training / 451 testing) — public medical VQA benchmark

Baselines vs proposed

• PaliGemma 2 (3B): Report CIDEr = 0.6 vs RadGrounder 39.2
• Gemma 3 (4B): VQA Acc. = 53.0 (Slake) vs RadGrounder 88.1
• MedGemma (4B): VQA Acc. = 87.6 (Slake) vs RadGrounder 88.1
• BiomedGPT-B: VQA F1 = 85.2 (Slake) vs RadGrounder 87.7
• LLaVA-Med: VQA Acc. = 86.8 (Slake) vs RadGrounder 88.1
• Med-Gemini: VQA F1 = 75.8 (Slake) vs RadGrounder 87.7
• RadGrounder token-based detection G-IoU = 43.6 vs segmentation G-IoU = 36.9
• Adding internal RefRad2D data to external fine-tuning improves Slake open-ended VQA F1 from 86.9 to 87.7

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20477.

Fig 1: Overview of the components to generate the data for visual grounding. We use

Fig 2: Qualitative example of RadGrounder’s report generation on an abdominal CT

Fig 3: Detection grounding performance on a chest CT (lung window). RadGrounder

Fig 4 (page 9).

Limitations

• Data originates from a single university hospital, limiting multi-center generalization and robustness evaluations.
• Spatial grounding annotations cover anatomical structures but do not explicitly localize pathological findings or lesions, limiting clinical localization capabilities.
• Comparison between detection and segmentation grounding is difficult due to fundamentally different geometric tasks and evaluation metrics.
• Synthetic VQA pairs are generated automatically and may not capture all clinical complexity or real-world question distributions.
• No explicit adversarial or robustness evaluation against noisy or out-of-domain imaging conditions was conducted.
• LLMScore depends on a large proprietary LLM, which may limit reproducibility and external validation.

Open questions / follow-ons

• Can multi-center clinical data integration improve model generalization and robustness across institutions?
• How to extend spatial grounding from anatomy to pathological lesion-level localization enabling clinically actionable outputs?
• What are the benefits and tradeoffs of detection vs segmentation grounding approaches under varied clinical tasks and datasets?
• Could leveraging additional modalities (e.g., 3D volumetric context) further enhance spatial grounding and report accuracy?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, this work demonstrates a scalable approach to learning spatially grounded textual descriptions from complex clinical image data without manual annotations. Although focused on radiology, the methodology of combining large-scale automated data curation, multi-task learning, and grounding supervision could inspire scalable training pipelines for other vision-language tasks requiring spatial verification. The demonstration that spatial grounding supervision can be added without reducing language performance highlights promising strategies for verifiable and interpretable model outputs, which is relevant in contexts where trustworthiness and explanation are important, such as bot detection or human verification. The token-based bounding box generation approach also illustrates a novel way to embed spatial outputs in text generation frameworks, hinting at architectures that unify spatial and linguistic prediction seamlessly. However, direct transfers would require addressing domain differences and adversarial robustness in bot-defense contexts.

Cite

@article{arxiv2606_20477,
  title={ Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology },
  author={ Yusuf Salcan and Simon Ging and Robin Schirrmeister and Philipp Arnold and Elmar Kotter and Behzad Bozorgtabar and Thomas Brox },
  journal={arXiv preprint arXiv:2606.20477},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20477}
}

Read the full paper

• arXiv:2606.20477 (abstract)
• arXiv:2606.20477 (PDF)