Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models
Source: arXiv:2607.21496 · Published 2026-07-23 · By Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao
TL;DR
This paper addresses the critical problem of early and accurate cognitive impairment (CI) detection using non-invasive speech-based biomarkers. Speech contains linguistic and acoustic markers sensitive to cognitive decline, but prior approaches often rely on a single modality, limiting robustness and generalization across diverse speakers and recording conditions. Leveraging recent advances in large language models (LLMs), the authors propose a novel multimodal CI detection framework that integrates acoustic embeddings extracted directly from speech audio with linguistic embeddings derived from automatically transcribed text using open-source Qwen-family LLMs. This design enables privacy-preserving, local deployment compliant with healthcare regulations.
The framework concatenates modality-specific embeddings and feeds them into a downstream classifier, capturing complementary speech production and content cues. Evaluation on standard ADReSS20 and ADReSSo21 benchmark datasets shows the multimodal approach achieves 92.4% CI classification accuracy, outperforming single-modality baselines and prior state-of-the-art models by 7.5% absolute accuracy. The model demonstrates superior cross-dataset generalization, robustness to dataset shifts, and improved early detection capability, particularly for mild CI stages. This work establishes a new SOTA for speech-based CI identification by effectively exploiting the representational power of multimodal LLM embeddings while preserving patient privacy through local, open-source modeling.
Key findings
- The multimodal framework combining acoustic and linguistic embeddings achieves 92.4% CI classification accuracy on combined ADReSS20+ADReSSo21 test sets, surpassing prior top baseline by 7.5%.
- Neural network classifiers on fused embeddings outperform classical ML models at test time, achieving an F1-score of 92.2%, despite lower cross-validation stability.
- Acoustic embeddings are extracted from Qwen2.5-7B AudioLLM and linguistic embeddings from Qwen3-30B text LLM, concatenated and normalized before classification.
- Cross-dataset evaluation (training on one dataset and testing on the other) shows strong generalization to realistic population and recording shifts.
- The approach improves early-stage CI detection sensitivity, particularly within mild impairment MMSE range (19–23), indicating capacity to identify subtle cognitive changes.
- Precision reaches 94.6%, exceeding prior methods, reducing false positives critical for clinical screening applicability.
- The system only requires raw audio as input and uses automated ASR transcripts, avoiding manual annotation and preserving privacy via local deployment of open-source models.
- Preprocessing steps including resampling, amplitude normalization, voice activity detection, and fixed-length segmentation aid in robust feature extraction and model performance.
Threat model
The adversary is an unauthorized party attempting to access sensitive patient data used for cognitive impairment detection. The framework assumes local deployment with no raw data transmitted externally, mitigating risks of data leakage. Adversaries cannot interfere with model training or access raw speech or transcripts directly, only aggregated embeddings. The system does not defend against adversaries able to manipulate input speech at collection but prioritizes privacy by design to comply with healthcare data protection regulations.
Methodology — deep read
The authors design a privacy-preserving, generalizable multimodal CI detection pipeline leveraging speech audio and transcripts processed through large language models (LLMs). The core threat model assumes clinical usage where patient privacy is critical, so only anonymized embeddings are retained without access to raw transcripts or audio beyond local processing.
Two publicly available benchmark datasets—ADReSS20 and ADReSSo21—serve as experimental testbeds, with balanced cognitively impaired (CI) and cognitively normal (CN) participants performing a spontaneous speech task describing the Cookie Theft picture. ADReSS20 provides manual transcripts and segmented audio, while ADReSSo21 contains full audio only, introducing realistic dataset shifts concerning recording conditions and transcript availability.
Audio preprocessing includes resampling audio to fixed rates, downmixing to mono, amplitude normalization, voice activity detection (removing silences/background noise), and segmentation into fixed-length chunks for manageable processing.
Acoustic embeddings are extracted via a Qwen2.5-7B Audio Large Language Model (AudioLLM) applied to each audio segment, capturing spectral, temporal, and prosodic features relevant to cognitive decline. Segment embeddings are pooled by averaging over time to create a fixed-length subject-level acoustic embedding.
Parallelly, the segmented audio is fed into an automatic speech recognition (ASR) system to produce transcripts, concatenated and normalized before being tokenized and passed to a text-based Qwen3-30B Large Language Model to obtain high-dimensional linguistic embeddings capturing semantic, syntactic, and discourse information.
These modality-specific embeddings are concatenated into a unified multimodal vector and normalized with Min–Max scaling, mitigating scale disparities between acoustic and text embeddings.
The normalized multimodal embeddings form input to various classification models—support vector classifier (SVC), logistic regression, XGBoost, and multilayer neural networks—with hyperparameters optimized via grid search and evaluated under five-fold cross-validation and held-out testing. Neural nets, despite cross-validation variability, achieve the best test results due to their ability to model complex non-linear modality interactions.
To assess generalization, two experimental configurations are used: 1) combined training and testing on merged datasets, and 2) cross-dataset evaluation training on one and testing on the other without fine-tuning.
Performance metrics include accuracy, precision, recall, and F1-score, compared to representative prior speech-based CI detection approaches. Ablation studies isolate the impact of multimodal fusion versus single modalities, while MMSE score distributions are analyzed to examine early detection capability.
The authors commit to open-sourcing their code and models to facilitate reproducibility and real-world clinical adoption accounting for privacy compliance.
Technical innovations
- Integration of acoustic embeddings from an AudioLLM (Qwen2.5-7B) with linguistic embeddings from a large text LLM (Qwen3-30B) via feature concatenation to create a unified multimodal representation for CI detection.
- Development of a fully local, privacy-preserving pipeline that avoids external cloud APIs by using open-source LLMs for both ASR transcription and embedding extraction.
- Use of Min–Max normalization on concatenated multimodal embeddings to stabilize disparate modality scales and enhance classifier training and robustness.
- Demonstration of strong cross-dataset generalization on heterogeneous clinical speech datasets through evaluation on ADReSS20 and ADReSSo21 with realistic dataset shifts.
- Application of neural networks to exploit high-dimensional, fused multimodal embeddings effectively, enabling superior modeling of non-linear cross-modal interactions.
Datasets
- ADReSS20 — 156 participants (108 train, 48 test) balanced CI and CN — public benchmark dataset for speech-based cognitive impairment detection
- ADReSSo21 — 237 participants (166 train, 71 test) balanced CI and CN — public benchmark dataset with differing audio/transcript formats from ADReSS20
Baselines vs proposed
- Mortensen et al. (Zephyr + XGBoost): Accuracy = 84.9% vs Proposed: 92.4%
- Bang et al. (BERT): Accuracy = 83.1% vs Proposed: 92.4%
- Agbavor et al. (GPT-3.5 + SVC): Accuracy = 80.3%, Precision = 72.3%, Recall = 97.1%, F1 = 82.9% vs Proposed: Acc 92.4%, Prec 94.6%, Rec 89.8%, F1 92.2%
- Luz et al. (SVC): Accuracy = 78.9% vs Proposed: 92.4%
- Neural Networks on fused embeddings outperform SVC (Accuracy 89.9%) and XGBoost (Accuracy 84.9%) with final test accuracy 92.4%.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21496.
Fig 1: The workflow of the proposed framework.
Fig 2: presents the distribution of MMSE scores for CI cases correctly identified by the proposed multimodal
Fig 3: ROC curve comparing multimodal and single modalities with 5-fold cross validation derived AUC standard deviations.
Limitations
- Evaluation limited to two English datasets (ADReSS20 and ADReSSo21) describing the same speech task, limiting language and task diversity.
- Use of a single ASR pipeline may introduce transcription errors affecting linguistic embeddings and downstream classification.
- Neural network classifier shows high variance and requires larger datasets for more stable training; applicability to low-resource settings unclear.
- Cross-dataset evaluation lacks fine-tuning, which may not fully reflect real-world domain adaptation possibilities or challenges.
- No explicit adversarial robustness evaluation or testing under noisy or highly heterogeneous clinical audio settings.
- Limited analysis on interpretability of embeddings or explainability of model decisions for clinical practitioners.
Open questions / follow-ons
- How would multimodal LLM-based CI detection perform across languages, dialects, and clinically distinct populations beyond the evaluated English datasets?
- Can ASR errors or transcription uncertainties be better modeled to improve robustness and reduce false negatives in early CI detection?
- How does the method perform under different recording environments, noise levels, or device variability affecting acoustic embeddings?
- What interpretability methods can be applied to multimodal embeddings to enhance clinical trust and understanding of model decisions?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this paper demonstrates the power of multimodal fusion of linguistic and acoustic signals via large language models for detecting subtle behavioral patterns—in this case, cognitive impairment from speech. The methodology highlights how integrating multiple complementary data streams with LLM embeddings and employing normalization enables robust classification across realistic real-world heterogeneity and dataset shifts.
While not directly related to bot detection, the technical insights into privacy-preserving local deployment of LLMs, strategic multimodal embedding fusion, and cross-domain generalization may inspire analogous approaches for behavioral biometrics or human interaction analysis to distinguish automated from genuine users. The work underscores the importance of combining multiple weak behavioral signals to improve detection sensitivity without compromising privacy, a theme relevant in scalable anti-bot systems.
Cite
@article{arxiv2607_21496,
title={ Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models },
author={ Yingchao Huang and Xin Wang and Yuhan Su and Shanshan Yao },
journal={arXiv preprint arXiv:2607.21496},
year={ 2026 },
url={https://arxiv.org/abs/2607.21496}
}