ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Source: arXiv:2607.24743 · Published 2026-07-27 · By Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang et al.

TL;DR

ClinFusion addresses critical limitations in applying multimodal large language models (MLLMs) to clinical settings by focusing on a vision-centric approach that unifies 2D and native 3D medical image understanding. It introduces a compositional and cascaded vision encoder architecture with a novel Cascade Spatial-Aware Locality Fusion operator that progressively enriches visual representations by fusing specialized 2D and a dedicated 3D encoder alongside a foundational visual transformer. This design enables enhanced absorption of heterogeneous medical visual knowledge beyond prior monolithic or parallel encoder approaches.

ClinFusion also pioneers a vision-grounded evaluation framework with two major contributions: MedIF-Bench, a benchmark assessing instruction-following in medical contexts crucial for clinical interaction, and a region-of-interest (RoI)-grounded report generation evaluation that conditions report assessment on clinically relevant anatomical context. This method breaks down diagnostic claims into matched, missed, and hallucinated findings with an LLM-as-a-judge, producing the strongest correlations with board-certified radiologists' blinded evaluations. ClinFusion outperforms a broad set of open-source and proprietary medical MLLMs across 2D/3D VQA, report generation, and textual medical tasks, demonstrating state-of-the-art performance validated both quantitatively and via expert clinical review.

Key findings

• ClinFusion-8B surpasses Hulu-Med-7B by 14.5 points on AMOS-MCQ (80.2 vs 65.7) and outperforms Gemini-3-Flash by 16 points on the same 3D volumetric VQA task.
• On 2D clinical report generation, ClinFusion-8B achieves F1 scores of 37.8 (CheXpert-Plus) and 57.3 (IU-XRAY), substantially exceeding Hulu-Med-7B (31.9 and 46.5 respectively).
• ClinFusion-32B achieves best reported F1 of 23.9 on CT-Rate Report, surpassing Hulu-Med-32B (23.4) and proprietary models such as Gemini-3-Flash (20.2) and GPT-5.2 (14.1).
• On instruction-following (MedIF-Bench), ClinFusion-32B obtains 98.9% compliance, outperforming GPT-5.2 (96.0), Gemini-3-Flash (96.6), and other medical MLLMs that suffer degradation from domain adaptation.
• ClinFusion-32B achieves highest accuracy among medical MLLMs on seven of eight textual medical benchmarks, e.g., MedXpertQA (26.7 vs Hulu-Med-32B 19.8) and MedQA-MCMLE (93.8 vs 87.0).
• Blinded evaluation by six board-certified radiologists on 300 cases ranking reports shows ClinFusion with agentic tools ranked best in Factual Accuracy, Completeness, and Clinical Utility with p < 0.001 against Hulu-Med and Gemini-3-Flash.
• The proposed RoI-grounded evaluation metric achieves top correlation with expert judgment across all measures (Kendall’s τ=0.511, Spearman’s ρ=0.572, Top-1 Accuracy=55.5%), outperforming 10 other automatic metrics.
• Incorporating clinical context in report evaluation improves metric sensitivity and separates hallucinated versus plausible findings, preventing score inflation from off-target text generation.

Threat model

The adversary is implicitly modeled as an insufficient medical AI system that may fail to interpret heterogeneous 2D/3D medical images or follow complex clinical instructions accurately, risking hallucination, omission, or misdiagnosis. The model must handle diverse imaging modalities and align outputs with clinical context to be useful. There is no explicit malicious adversary or attack scenario considered—e.g., no data poisoning or adversarial examples are evaluated.

Methodology — deep read

1. Threat Model & Assumptions: ClinFusion targets clinical deployment with adversaries modeled as medical AI systems needing to robustly interpret heterogeneous 2D and 3D medical imaging under clinical constraints. The assumption is that models must correctly follow complex medical instructions and produce factual, contextually grounded diagnostic reports; adversaries could represent inadequate or incomplete models prone to hallucination or context-agnostic generation. No explicit adversarial attack scenario is considered.

2. Data: ClinFusion is trained and evaluated on multiple publicly available and proprietary multimodal medical datasets encompassing 2D images (X-rays, pathology slides, fundus photos) and 3D volumes (CT, MRI scans). 3D volumes are processed with 32 slices sampled per volume at 256×256 resolution. Dataset splits, preprocessing details, and label curation align with benchmark protocols including CheXpert-Plus, IU-XRAY, AMOS-MM, 3D-RAD, and a custom MedIF-Bench instruction-following set. Labels include visual question answers, structured clinical reports, and instruction compliance.

3. Architecture: ClinFusion’s core is a compositional vision encoder combining a foundational Qwen ViT transformer with an ensemble of specialist 2D encoders cascaded through the Cascade Spatial-Aware Locality (CaSL) Fusion operator. This hierarchical operator enables progressive feature enrichment by allowing cascaded interaction among 2D encoders, preserving alignment with the foundational representation. A dedicated 3D encoder is integrated for volumetric data, using a 2D-anchored depth-aware CaSL Fusion to guide 3D features by well-aligned 2D representations. This novel fusion contrasts with prior parallel multi-encoder schemes by enabling cascaded, spatially aware interactions and native 3D processing within the unified encoder. The enriched vision embedding is input to a large language model (Qwen-3-VL) for multimodal understanding and generation.

4. Training: Training details include standard fine-tuning of the composite vision-language model on curated multimodal clinical datasets, with hyperparameters and epochs unspecified in detail in the truncated text. Scaling experiments with models having 8B and 32B parameters demonstrate improvement with scale. Multiple seeds or hardware details are not explicitly reported. The vision and language components are jointly optimized for downstream tasks.

5. Evaluation Protocol: ClinFusion is evaluated against three model categories—generalist MLLMs (e.g., Qwen2.5-VL), open-source medical MLLMs (e.g., Hulu-Med, Lingshu), and proprietary models (GPT-5.2, Gemini-3-Flash, Claude-Sonnet-4.5). For 3D data, 32 slices per volume are used unless otherwise mandated. Metrics include accuracy for VQA tasks, and a novel RoI-grounded evaluation protocol for report generation, employing an LLM-as-a-Judge to score clinical factuality decomposed into matched, missed, and hallucinated findings within clinically relevant anatomical regions. Instruction-following is measured by MedIF-Bench using regex-based compliance. Statistical significance is established via blinded expert reader studies with p-values reported (<0.001). Inter-annotator agreement is measured with Kendall’s W.

6. Reproducibility: The authors provide open-source code and models on GitHub and Huggingface, supporting reproducibility. However, some datasets and detailed training configurations are proprietary or unspecified. The vision encoders are based on publicly released Qwen-VL series backbones.

Example End-to-End: For a chest CT case, the model inputs 32 volumetric slices processed through the 3D encoder with 2D-anchored CaSL Fusion. The enriched visual features are passed to the Qwen-3-VL LLM, which generates a detailed report focusing on specified anatomical regions (“Areas of Focus”). An expert radiologist then scores the report alongside comparator models, verifying accuracy and clinical relevance. The RoI-grounded metric corroborates this expert ranking by parsing the report claims into matched/missed/falsified findings within the patient’s clinical context.

Technical innovations

• Cascade Spatial-Aware Locality (CaSL) Fusion operator for cascaded interplay among multiple 2D vision encoders and 2D-anchored fusion with a native 3D encoder.
• Compositional vision encoder architecture that integrates foundational ViT with specialist encoders for heterogeneous 2D and 3D medical image understanding.
• Vision-grounded evaluation framework featuring MedIF-Bench for instruction-following and an RoI-grounded report generation metric conditioned on clinical context with LLM-as-a-judge deconstruction of findings.
• Agentic tool use extension combining retrieval-augmented generation and invoking specialized AI expert tools to enhance clinical workflows dynamically.

Datasets

• CheXpert-Plus — size unspecified — public medical chest X-ray dataset
• IU-XRAY — size unspecified — public chest X-ray dataset with reports
• AMOS-MM — size unspecified — abdominal multi-organ CT dataset
• 3D-RAD — size unspecified — CT radiology dataset
• MedIF-Bench — curated instruction-following benchmark in medical scenarios (introduced by authors, not publicly detailed in size)

Baselines vs proposed

• Hulu-Med-7B: AMOS-MCQ accuracy = 65.7 vs ClinFusion-8B = 80.2
• Hulu-Med-32B: AMOS-MCQ accuracy = 73.9 vs ClinFusion-8B = 80.2
• Gemini-3-Flash: AMOS-MCQ accuracy = 64.2 (inferred) vs ClinFusion-8B = 80.2
• Hulu-Med-7B: CheXpert-Plus report F1 = 31.9 vs ClinFusion-8B = 37.8
• Hulu-Med-7B: IU-XRAY report F1 = 46.5 vs ClinFusion-8B = 57.3
• Hulu-Med-32B: CT-Rate Report F1 = 23.4 vs ClinFusion-32B = 23.9
• Gemini-3-Flash: CT-Rate Report F1 = 20.2 vs ClinFusion-32B = 23.9
• MedGemma-1.5-4B-IT: Instruction-following Overall-IF = 27.4 vs ClinFusion-32B = 98.9
• GPT-5.2: Instruction-following Overall-IF = 96.0 vs ClinFusion-32B = 98.9
• ClinFusion-32B: MedXpertQA accuracy = 26.7 vs Hulu-Med-32B = 19.8

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24743.

Fig 1: Overview of the ClinFusion framework. a, Compositional vision encoder. We address the limitation of monolithic

Fig 2: The architecture details of ClinFusion, a multimodal large language model for holistic medical understanding. a, The

Fig 3: Performance of ClinFusion on 2D multimodal medical benchmarks. a, Results on 2D multimodal medical VQA

Fig 4 (page 30).

Fig 4: Performance of ClinFusion on 3D multimodal medical benchmarks and MedIF-Bench. a, 3D multimodal medical

Fig 6 (page 32).

Fig 5: Expert radiologist evaluation.

Fig 8 (page 34).

Limitations

• Authors do not detail specific adversarial robustness or attack evaluations, so resilience to malicious clinical image manipulation is untested.
• Some datasets and task configurations are proprietary or underspecified in training detail, potentially impacting full reproducibility.
• 3D volumetric inputs use fixed slice sampling (32 slices at 256×256) which may discard some 3D contextual information in complex cases.
• The model shows gaps relative to strongest proprietary models on hardest reasoning tasks (e.g., MedXpertQA and SuperGPQA), though scaling helps mitigate this.
• Agentic tool use system design and integration details are briefly discussed but their comprehensive impact on clinical deployment is not fully explored.
• No explicit evaluation under realistic deployment constraints such as inference latency, hardware cost, or model interpretability is presented.

Open questions / follow-ons

• How robust is the CaSL Fusion approach under adversarial image perturbations, corrupted or missing slices, or rare pathology presentations?
• Can the agentic tool use framework be extended to systematically integrate multimodal clinical context data beyond imaging (e.g., EHR text, lab results) to improve diagnostic accuracy?
• What are the impacts of further scaling both vision and language components on closing remaining performance gaps against proprietary models on reasoning-heavy medical benchmarks?
• How will the RoI-grounded evaluation framework perform across other medical domains and specialties beyond radiology, such as pathology or dermatology?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, ClinFusion’s approach highlights the importance of developing vision-language fusion architectures that natively handle heterogeneous high-dimensional visual data—in this case, 2D and volumetric 3D medical images—which may analogously inform systems that must verify or authenticate complex visual inputs under adversarial conditions. The vision-grounded evaluation methodology exemplifies how domain-specific context-aware evaluation aligned with expert human judgment can provide more meaningful assessments than simple text matching, pointing to the benefit of task- and context-specific benchmarks for multimodal systems.

While ClinFusion is tailored for high-stakes clinical use rather than bot detection, the principles of progressive cascaded fusion, alignment of multimodal encoders, and decomposing output fidelity by region or semantic relevance could inspire more sophisticated CAPTCHA or bot-defense systems that analyze heterogeneous visual inputs. Moreover, the use of agentic tool use to dynamically augment model knowledge and perception capabilities might motivate enhanced multi-agent or multi-tool frameworks for bot detection integrating disparate signals.

Cite

@article{arxiv2607_24743,
  title={ ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding },
  author={ Hangjie Yuan and Yichen Qian and Zhiwei Tang and Xianzhe Xu and Lirong Wu and Sicheng Yang and Jinwang Wang and Pengju Wang and Zhitao Zeng and Yizeng Han and Yan Xing and Shengxuan Luo and Tao Feng and Qing Xie and Weigen Yao and Yi Yang and Zuozhu Liu and Jiasheng Tang and Shaocheng Wang and Jitao Wang and Jiahong Dong and Weihua Chen and Feng Xu and Fan Wang },
  journal={arXiv preprint arXiv:2607.24743},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24743}
}

Read the full paper

• arXiv:2607.24743 (abstract)
• arXiv:2607.24743 (PDF)