MyoCardBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios
Source: arXiv:2607.25186 · Published 2026-07-28 · By Xiao Li, Mouxiao Bian, Zhaodi Wu, Sijie Ren, Juechen Chen, Lu Lu et al.
TL;DR
MyoCardBench addresses a critical gap in evaluating medical large language models (LLMs) by providing a comprehensive, real-world benchmark spanning the cardiovascular care continuum. Unlike prior benchmarks focused on isolated exam-style questions or single modalities, MyoCardBench covers 13 clinically authentic task-specific datasets derived from de-identified cardiovascular records, including documentation, diagnostic reasoning, risk scoring, treatment planning, emergency management, multimodal image interpretation, longitudinal care, communication, and ethics. Expert cardiologists with intermediate and senior titles rigorously annotated task inputs, generated clinically sufficient reference answers, and established atomic key-point standards for granular evaluation. Seven state-of-the-art LLMs were evaluated in a zero-shot setting, generating over 15,000 responses scored with complementary metrics assessing explicit key-point recall and holistic clinical quality.
The study found GPT-5.4 achieved the highest overall macro-average score (62.55/100) and item-weighted mean (62.19), ranking first across the three evaluated dimensions: full-cycle cardiovascular care, multimodal interpretation and documentation, and communication, safety and ethics. However, even the strongest model’s average reflects substantial room for clinical improvement. Among tasks, auxiliary report integration and chronic medication management proved most tractable (cross-model means >70), while ECG interpretation and clinical ethics were the most challenging (≈17). There was notable divergence between key-point coverage and holistic quality scores for tasks requiring complex prioritization and semantic synthesis (e.g., communication, emergency rescue, treatment planning), illustrating that fluent responses may omit clinically critical elements. Benchmark insights highlight LLM strengths and weaknesses with task granularity and support future model development focusing on clinically essential omissions and specialties.
Key findings
- MyoCardBench includes 2,263 clinically authentic items across 13 cardiovascular-specialty tasks covering longitudinal care, multimodal inputs, communication, and ethics.
- Seven LLMs generated 15,841 zero-shot outputs evaluated with dual metrics: atomic key-point macro-recall and holistic clinical quality scored 0-100.
- GPT-5.4 achieved the highest overall macro-average score of 62.55 and item-weighted mean of 62.19, outperforming Gemini 3.1 Pro (59.95) and Qwen 3.6 27B (59.72).
- GPT-5.4 ranked first within all three clinical dimensions: full-cycle care (70.42), multimodal interpretation/documentation (58.47), and communication/safety/ethics (43.17).
- Highest cross-model task performance was seen in CardioAuxReport (86.38), CardioChronicMed (75.04), and CardioAdmRec (74.66).
- Lowest cross-model means were in CardioECGRead (17.25) and CardioEthics (17.34), indicating these as notably difficult tasks across all LLMs.
- Divergence between holistic clinical quality and key-point coverage reached up to 52.71 points in CardioComm, suggesting fluent output can omit important clinical facts.
- Task-level leader counts and rankings showed no LLM dominated all tasks; GPT-5.4 led six tasks while other models led one to four tasks, revealing domain specialization among LLMs.
Threat model
Not applicable, as the study focuses on clinically authentic multi-task evaluation of large language models rather than adversarial attacks or security threats. The benchmark assumes LLMs operate with no external auxiliary tools or memory and are evaluated on their raw reasoning and generation capabilities.
Methodology — deep read
Threat Model & Assumptions: The benchmark targets LLMs used for clinical decision support in cardiovascular care, assuming zero-shot inference without access to external retrieval, calculators, or conversation history. Adversarial threat modeling was not the focus; rather, the study aims to evaluate clinical competence and safety-critical reasoning.
Data: MyoCardBench comprises 2,263 items from de-identified real-world cardiovascular clinical records supplied by Zhongshan Hospital, Fudan University, representing inpatient and outpatient encounters with diverse input types—textual history, labs, imaging (ECG and coronary CTA), longitudinal follow-ups, communication, and ethics cases. Items were selected to ensure sufficient information for the target task without needing invented facts. Duplicate or ambiguous cases were excluded. The dataset includes 2,048 open-ended responses and 215 multiple-choice ethics questions.
Architecture/Algorithm: Seven LLMs, including GPT-5.4, Gemini 3.1 Pro, Qwen 3.6 27B, Grok 4.3, and others, were evaluated in zero-shot, single-turn inference using deterministic decoding (temperature=0) on identical clinical prompts per item. Multimodal inputs (images plus text) were provided where relevant. No chain-of-thought rationale or external plugins were used.
Training Regime: Not applicable, as the study did not train models but performed inference/evaluation on fixed released LLM versions in early/mid 2026.
Evaluation Protocol: Two complementary metrics assessed open-ended items: key-point macro-recall quantifying recovery of clinician-defined atomic facts (based on structured reference key points), and holistic clinical quality scored by expert cardiologists judging clinical correctness, completeness, safety, and organization. A composite score weighed these two components. The ethics dataset used accuracy on multiple-choice answers. Aggregate statistics included task-level macro-averages, item-weighted means, and inter-model performance ranges. No patient-level clustering or formal statistical significance testing was performed because only aggregate data were available.
Reproducibility: The dataset is derived from a clinical institution and contains protected health information, so it is not publicly available. LLM models are commercially or internally released without open weights except Qwen 3.6 and 3.5. Analysis code and summary statistics were provided to support reproducibility, but item-level predictions and source data are closed.
Technical innovations
- Development of a large-scale, longitudinal, multimodal, and multi-task benchmark specifically tailored to cardiovascular specialist care, covering documentation, diagnosis, emergency, longitudinal management, communication, and ethics.
- Use of a dual-metric evaluation framework combining atomic key-point macro-recall with holistic clinical-quality scoring to capture both explicit clinical content coverage and overall response plausibility.
- Expert annotation protocol involving intermediate-level cardiologists for initial reference and key-point creation, followed by senior cardiologist cross-review and adjudication to enhance clinical validity and consistency.
- Zero-shot inference evaluation strictly controlling input prompts, decoding parameters, and multimodal data formats to enable comparative assessment across leading 2026 large language models.
Datasets
- MyoCardBench — 2,263 items across 13 specialized cardiovascular clinical datasets — de-identified real-world records from Zhongshan Hospital, Fudan University
Baselines vs proposed
- GPT-5.4: overall macro-average score = 62.55 vs Gemini 3.1 Pro = 59.95 vs Qwen 3.6 27B = 59.72
- CardioAuxReport task: cross-model mean key-point coverage = 86.38 (highest among all tasks)
- CardioECGRead task: cross-model mean key-point coverage = 17.25 (lowest among all tasks)
- CardioEthics task: accuracy = 17.34 (lowest among all tasks)
- GPT-5.4: full-cycle care dimension score = 70.42 vs Gemini 3.1 Pro = 67.38 vs Qwen 3.6 27B = 67.10
- GPT-5.4: communication, safety, and ethics dimension score = 43.17 vs Gemini 3.1 Pro = 37.41 vs Qwen 3.6 27B = 41.21
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.25186.
Fig 1: Overview of the MyoCardBench construction and evaluation framework
Fig 2: Dimension-specific and overall model performance
Fig 3: Task-level performance and between-LLM variability across MyoCardBench
Fig 4: Divergence between holistic clinical quality and key-point coverage
Fig 5: LLM-specific strengths and performance stability
Limitations
- The benchmark covers only cardiovascular care and may not generalize to other medical specialties or workflows.
- Data is derived from a single hospital system and geographic region (Shanghai, China), which may limit diversity of clinical presentations and external validity.
- Evaluation used zero-shot LLM prompting without few-shot or retrieval-augmented strategies, so real-world clinical deployments with additional context may yield different results.
- Assessment metrics are based on expert-generated key points and clinical rubrics; subjective elements in holistic scoring and interrater agreement were not detailed.
- Closed access to raw data and item-level outputs limits independent external validation and error analysis.
- No formal adversarial robustness testing or simulation of intentional manipulations impacting patient safety was performed.
Open questions / follow-ons
- How would inclusion of retrieval-augmented generation or few-shot learning impact model performance and reliability on MyoCardBench tasks?
- Can enhanced multimodal fusion techniques or domain-specific training improve low-performing tasks like ECG interpretation and clinical ethics reasoning?
- What level of interrater agreement exists for the holistic clinical quality scoring, and how can evaluation be standardized across diverse clinician raters?
- How well do these models maintain clinical safety and decision consistency under temporal distribution shifts or longitudinal patient tracking scenarios?
Why it matters for bot defense
MyoCardBench offers a rigorous framework for bot-defense and CAPTCHA practitioners interested in evaluating large language models’ ability to perform complex, safety-critical tasks under real-world clinical constraints. The benchmark’s multipronged assessment best models scenarios requiring integration of multimodal data, temporal reasoning, and nuanced human communication—all relevant to distinguishing authentic expert-like human interaction from automated or malicious bots.
Practitioners building CAPTCHA or bot-defense systems for healthcare settings can leverage insights from this study on model strengths and weaknesses, particularly in safely handling multimodal inputs and generating consistent, clinically valid outputs. The divergence found between key-point recall and holistic quality highlights risks of over-reliance on surface fluency, guiding stricter validation strategies to detect incomplete or unsafe LLM answers during deployment.
Cite
@article{arxiv2607_25186,
title={ MyoCardBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios },
author={ Xiao Li and Mouxiao Bian and Zhaodi Wu and Sijie Ren and Juechen Chen and Lu Lu and Jingru Ding and Yun Zhong and Jie Xu and Yixiu Liang and Junbo Ge },
journal={arXiv preprint arXiv:2607.25186},
year={ 2026 },
url={https://arxiv.org/abs/2607.25186}
}