Parameter-efficient Prompt Tuning of Vision Foundation Model With Adaptive Focal Loss for Interpretable MCI Screening

Source: arXiv:2607.15047 · Published 2026-07-16 · By Javad Khoramdel, Farhad Hoseyni, Amirhossein Nikoofard

TL;DR

This paper addresses the automated detection of Mild Cognitive Impairment (MCI), a subtle early cognitive decline often preceding Alzheimer's disease, from multimodal neuropsychological drawing tests. The main challenges include data scarcity, class imbalance, and diagnostic uncertainty near clinical score boundaries. Unlike existing methods that fully fine-tune large hybrid architectures and rely on post-hoc interpretability, the authors propose a parameter-efficient framework that leverages a frozen DINOv2-Small vision transformer backbone, adapted using three modality-specific learnable prompt tokens interacting with image patch tokens via cross-attention. This design intrinsically produces spatial attention maps providing native interpretability. Additionally, a novel MoCA-adapted focal loss integrates continuous cognitive scores into target labels, loss modulation, and adaptive weighting, explicitly modeling uncertainty near clinical cutoffs. Under five-fold stratified cross-validation on a public triplet drawing dataset, the method achieves a mean MCI-class F1 score of 0.641 and AUC of 0.795, outperforming a computationally heavier ResViT baseline by 0.110 in F1. Complementary augmentation and sampling strategies further improve performance and robustness.

Key findings

• Proposed method achieves mean MCI-class F1 of 0.641 ± 0.026 and AUC-ROC of 0.795 ± 0.024 across 5-fold CV on 918-subject neuropsychological drawing dataset.
• Outperforms ResViT baseline by +0.110 in MCI-class F1 and +0.054 in recall under identical training and evaluation protocols.
• Task-specific prompt tuning adapts frozen DINOv2 backbone with only 1.19 million trainable parameters (~6% of full network), significantly reducing compute versus full fine-tuning.
• Cross-attention saliency maps provide intrinsic spatial explainability identifying clinically relevant drawing regions associated with cognitive impairment.
• MoCA-adapted focal loss incorporating continuous cognitive scores improves model calibration near diagnostic cutoff, outperforming hard-label and prior soft-label baselines.
• Targeted augmentations (image inversion, class-balanced sampling, type-preserving Mixup, drawing swap) each independently increase MCI-class F1 by up to +0.022.
• Learnable attention weights over modalities (clock, cube, trail) provide clinically coherent modality importance insights per patient.
• Ablation studies confirm complementarity of architectural components and loss design; removing either MoCA-probability modulator or adaptive per-sample weighting degrades F1 by 0.017 to 0.021.

Threat model

The adversary is a non-adaptive, naturally occurring distribution of neuropsychological drawings exhibiting class imbalance and inherent diagnostic ambiguity near the MoCA score cutoff. The model must perform robust MCI detection despite label noise and limited data diversity. No explicit adversarial manipulations or evasion attacks are considered.

Methodology — deep read

The threat model assumes automated screening of MCI patients from triplet neuropsychological drawing inputs (clock, cube, trail), with unknown clinical labels used as ground truth. The adversary is not explicitly modeled; focus is on robust classification despite class imbalance and boundary uncertainty.

Data originates from a publicly available dataset with 918 elderly subjects (651 healthy controls, 267 MCI) collected at King Chulalongkorn Memorial Hospital. Each subject contributed 3 digitized drawing images and a continuous MoCA cognitive score. Labels are binary based on MoCA cutoff at 25. Five-fold stratified cross-validation is used to preserve class ratio. Standard augmentations plus domain-specific image inversion, class-balanced sampling, drawing swap, and modality-specific Mixup are applied.

Architecture centers on a frozen DINOv2-Small vision transformer backbone pretrained via self-supervised learning. Three learnable continuous prompt tokens, one per drawing modality, are introduced and serve as queries in a shared cross-attention layer attending over the frozen final image patch tokens per modality. This yields three embeddings for clock, cube, and trail images, respectively. A second learnable query attention layer aggregates these embeddings into a fused representation conveying modality importance.

Classification is performed by a linear head on the fused vector for binary MCI screening. Spatial attention maps from cross-attention weights provide intrinsic explainability by identifying which patches contribute most per modality.

Training uses AdamW optimizer for 40 epochs with batch size 4, learning rate 1e-4, weight decay 0.05, and warm-up followed by cosine annealing. The primary loss is a novel MoCA-adapted focal loss that leverages continuous MoCA scores to define soft targets, focal loss reweighting tied to MoCA-prediction misalignment, and adaptive per-sample weighting inversely proportional to MoCA score bin frequency.

Evaluation metrics include mean and standard deviation of MCI-class F1 (primary metric), ROC-AUC, accuracy, macro-F1, MCI-class precision, and recall. Checkpoints were selected by best fold validation MCI F1. Baselines include a re-implementation of the heavier ResViT model and variants of the loss function.

The paper ablates loss components (soft target, MoCA-probability modulator, adaptive weighting), architectural prompt designs, and augmentation/sampling strategies to isolate their contributions. Qualitative attention map visualizations confirm clinical alignment of learned features.

Code and pretrained weights are not reported as released; the dataset is publicly available through prior work [2]. Overall, the approach combines parameter-efficient prompt tuning, interpretable attention mechanisms, and cognitively-informed loss functions to improve MCI detection under data and label constraints inherent to clinical neuropsychological testing.

Technical innovations

• Modality-specific learnable prompt tokens operating as queries in a shared cross-attention layer over frozen vision foundation model patch tokens to enable parameter-efficient multimodal adaptation.
• MoCA-adapted focal loss that tightly couples continuous cognitive scores with target labels, loss modulation, and adaptive per-sample weighting to model diagnostic uncertainty near clinical cutoff.
• Learnable query attention fusion module aggregating modality-specific embeddings into interpretable patient-level modality importance weights.
• Targeted data augmentation including type-preserving Mixup and MoCA-neighbour drawing swap to enhance multimodal diversity and robustness while preserving clinical semantics.

Datasets

• Multi-drawing MCI dataset — 918 subjects with 2,754 images total (three drawings per subject: clock, cube, trail) — public via Ruengchaijatuporn et al. [2]

Baselines vs proposed

• ResViT baseline: MCI-class F1 = 0.531 ± 0.045 vs proposed: 0.641 ± 0.026
• ResViT baseline: MCI-class recall = 0.513 ± 0.116 vs proposed: 0.719 ± 0.096
• MoCA Soft Target [2] baseline on same architecture: AUC-ROC = 0.803 ± 0.027 vs full system: 0.795 ± 0.024
• Type-preserving Mixup augmentation improves MCI-class F1 from 0.617 ± 0.055 to 0.639 ± 0.030 (+0.022)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15047.

Fig 1: Overview of the proposed framework. Each drawing (clock, cube, trail) is processed by a shared frozen

Fig 2: Class distribution of the full dataset. Of 918

Fig 3: MoCA-score distribution across the dataset,

Fig 4: Cross-attention maps and learnable pool weights for two representative subjects. Each column shows one

Fig 5 (page 8).

Fig 6 (page 8).

Fig 7 (page 8).

Fig 8 (page 8).

Limitations

• Dataset is modest in size (918 subjects) and collected from a single institution, potentially limiting generalizability.
• No adversarial robustness or distribution shift evaluations are reported, e.g., for unseen scanners or populations.
• False positives due to subtle stylistic drawing variations remain a challenge, indicating limits of purely image-based MCI screening.
• No reported public release of code or pretrained weights, potentially hindering reproducibility.
• The patient label boundary remains fixed at MoCA=25, despite ongoing clinical ambiguity in cognitive decline thresholds.
• Evaluation focuses on cross-validation within a single dataset; external validation data is absent.

Open questions / follow-ons

• Can the parameter-efficient prompt tuning approach generalize to larger cohorts or external datasets with differing demographic distributions?
• How resilient are cross-attention based interpretability signals to adversarially perturbed or distorted inputs?
• What is the impact of alternative continuous cognitive score modeling approaches or dynamic clinical cutoffs on performance and calibration?
• Can multimodal extensions incorporating demographic or clinical metadata alongside drawings further improve diagnostic accuracy?

Why it matters for bot defense

This work exemplifies parameter-efficient adaptation of large frozen vision foundation models using modality-specific prompts, a paradigm that can inspire more efficient and interpretable bot-detection or CAPTCHA classification systems where multi-input modalities or limited labeled data exist. The intrinsic spatial explainability afforded by the cross-attention mechanism provides actionable interpretability without costly post-hoc methods, valuable for security engineers requiring model transparency against adversarial abuse. Furthermore, the proposed cognitive-score-informed loss modulator anticipates boundary uncertainty and label ambiguity—analogous to noisy signals in bot detection—suggesting adaptive loss designs that prioritize uncertain or borderline samples could improve robustness. Although domain-specific to neuropsychological screening, the model design and training strategies illuminate general principles applicable to bot defense systems constrained by data scarcity and requiring fine-grained interpretability. Practitioners should consider similar prompt-token schemes and attention fusion techniques to achieve parameter efficiency, modular multi-input handling, and inherent saliency attribution in CAPTCHA or bot classifiers.

Cite

@article{arxiv2607_15047,
  title={ Parameter-efficient Prompt Tuning of Vision Foundation Model With Adaptive Focal Loss for Interpretable MCI Screening },
  author={ Javad Khoramdel and Farhad Hoseyni and Amirhossein Nikoofard },
  journal={arXiv preprint arXiv:2607.15047},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15047}
}

Read the full paper

• arXiv:2607.15047 (abstract)
• arXiv:2607.15047 (PDF)