UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging

Source: arXiv:2607.21546 · Published 2026-07-23 · By Rafsan Jany, Shadab Tanjeed Ahmad, Ahsan Bulbul, Tahsinul Islam, Md Azam Hossain, Abu Raihan Mostofa Kamal

TL;DR

This paper tackles the problem of cross-modal knowledge transfer for medical image segmentation without requiring paired datasets. Multimodal learning generally improves segmentation by leveraging complementary information from different imaging modalities, but paired multi-modal clinical data is rare. Existing cross-modal distillation methods struggle with large modality gaps and noisy supervision from uncertain source predictions. To address this, the authors propose UnDA, an anchor-guided framework that aligns unpaired modalities through semantically structured class tokens extracted by an attention-based pooling module. A novel uncertainty-weighted optimal transport (UCT-OT) loss weighs alignment by prediction confidence to suppress noise, combined with a per-class ProtoNCE loss to maintain stable prototype memories and enforce global discriminability. UnDA is backbone-agnostic and removes the alignment module at inference, ensuring zero overhead.

They validate UnDA on two challenging cross-modal segmentation settings: brain tumor segmentation (T2-FLAIR to T1-native MRI) on the BraTS 2023 dataset and cardiac structure segmentation (MRI to CT) on MM-WHS, both under strictly unpaired training conditions. The results demonstrate consistent and substantial improvements over target-only baselines, significantly closing the performance gap to anchor modality references, particularly in Dice scores and boundary accuracy. Compared to recent state-of-the-art domain adaptation methods under a common backbone, UnDA achieves competitive performance with superior mean boundary delineation, highlighting its effectiveness in transferring meaningful structural features across heterogeneous modalities without paired data.

Key findings

• UnDA improves Dice scores on BraTS 2023 from target baseline 59.0% mean to 66.7%, recovering up to 36.7% of the gap to anchor modality (T2-FLAIR) performance.
• In brain tumor core segmentation, UnDA reduces Hausdorff Distance 95 (HD95) from 29.1mm to 12.4mm, enhancing boundary accuracy despite unpaired training.
• On MM-WHS cardiac dataset (MRI to CT), UnDA boosts mean Dice by +11.12 points over CT-only baseline (71.59% to 82.71%) and reduces HD95 by 7.47mm (21.41mm to 13.94mm).
• Uncertainty-weighted Optimal Transport (UCT-OT) suppresses noisy supervision by weighting anchor tokens by prediction confidence, outperforming unweighted OT which degrades performance.
• Prototype-level alignment with per-class ProtoNCE loss stabilizes feature alignment and improves performance on structures with high modal variability (e.g., right atrium and pulmonary artery).
• Compared under a common backbone on MM-WHS, UnDA achieves second highest mean Dice (82.71%) behind UMMKD (85.99%), but best mean HD95 (13.94mm), indicating robust boundary delineation.
• Ablation studies show combination of UCT-OT and ProtoNCE losses provides complementary gains, outperforming either alone.
• The class token attention pooling alignment module is backbone-agnostic and removed after training, imposing zero inference overhead.

Methodology — deep read

The objective is unpaired domain alignment for cross-modal knowledge transfer between an anchor modality (source) and a target modality (target) for medical image segmentation. The datasets are assumed unpaired with no spatial or patient correspondence. The anchor domain has labeled images DA={(XA_i, y_i)} and the target domain has labeled images DG={(XG_j, y_j)}, but samples are drawn independently. The goal is to train a target model benefiting from structural knowledge learned on anchor data, while relying only on target inputs at inference.

The framework is backbone-agnostic preventing coupling to specific network architectures or modality types. Both anchor and target encoders produce bottleneck feature maps. A lightweight alignment module sits at the bottleneck during training, converting the high-dimensional spatial feature maps into a fixed number K of semantically structured class tokens via attention-based pooling. This aligns bottleneck features into a shared class-token space to ease cross-modal correspondence despite heterogeneous inputs. Importantly, the alignment module is discarded at inference, incurring zero overhead.

The method trains in two phases. First, anchor training: the anchor encoder is trained on anchor domain images with supervised segmentation losses (Dice + BCE) plus an auxiliary class-presence loss supervising the class tokens. This produces reliable class-discriminative structural representations. Second, target training: the anchor model is frozen and used only for forward passes to generate reference tokens and uncertainty maps. The target encoder and a lightweight residual MLP adapter applied on its tokens are optimized with segmentation loss plus two key cross-modal alignment objectives.

Uncertainty-Weighted Optimal Transport (UCT-OT) computes an entropy-based uncertainty for anchor tokens from prediction entropy and uses it to weight the mass in the OT alignment, giving less weight to uncertain or noisy source tokens. OT finds an optimal transport plan minimizing alignment cost between anchor and target tokens subject to marginals defined by anchor token weights and uniform target distribution. Sinkhorn iterations are used to solve the entropy-regularized OT problem.

A prototype memory stores running exponential moving average class tokens from the anchor domain to stabilize batch-level alignment. The target tokens are encouraged with a ProtoNCE loss to cluster around these prototypes, enforcing discriminability and compactness in token space.

Segmentation is supervised by a weighted combination of Dice and BCE losses. Anchor training uses Lanchor = Lseg + λtoken Ltoken where Ltoken ensures class presence in tokens. Target training uses Ltarget = Lseg + λOT LUCT-OT + λP LProtoNCE, optimizing alignment and per-class feature consistency.

Data: Experiments use BraTS 2023 glioma MRI dataset (T2-FLAIR as anchor to T1-native target) with 1251 subjects split 80% train, 10% val, 10% test on disjoint patients; and MM-WHS cardiac dataset with MRI (anchor) and CT (target) unpaired volumes for seven cardiac structures. Both datasets underwent standard pre-processing (resampling, skull-stripping, normalization).

Training: Models based on 3D U-Net variants trained with AdamW optimizer, batch size 4, patch-based sampling, 250 epochs max with early stopping. Alignment module attached at encoder bottleneck only during training.

Evaluation: Metrics include mean Dice coefficient (%) and 95% Hausdorff distance (mm) over tumor/cardiac structures. Baselines: target-only supervised and fully supervised anchor. Also compared against state-of-the-art unpaired cross-modal segmentation methods retrained under the same backbone and protocol to ensure fair comparison.

Reproducibility: Implementation in PyTorch and MONAI available, uses public datasets. Some prior methods reimplemented with same backbone for benchmark comparison. Exact code release status not explicitly stated.

Technical innovations

• Backbone-agnostic alignment module that extracts semantically structured class tokens via attention-based pooling from encoder bottleneck features, removed at inference for zero overhead.
• Uncertainty-Weighted Optimal Transport (UCT-OT), a novel optimal transport formulation incorporating prediction uncertainty to dynamically weight source tokens and suppress noisy supervision during alignment.
• Per-class ProtoNCE loss combined with a running prototype memory to maintain global discriminability and enforce consistent class-level feature alignment across unpaired batches.
• Two-stage anchor-guided training scheme freezing a pretrained anchor model to generate stable alignment references while optimizing the target model on unpaired data.

Datasets

• BraTS 2023 Glioma — 1251 subjects — public brain tumor MRI dataset with T2-FLAIR and T1 modalities, unpaired splits used
• MM-WHS — dozens of MRI and CT cardiac volumes — public multi-modal cardiac segmentation dataset, modalities treated as unpaired

Baselines vs proposed

• BraTS 2023: Target (T1-native) baseline Dice mean = 59.0% vs UnDA 66.7%; HD95 mean 23.92mm vs UnDA 11.80mm
• MM-WHS: Target (CT) baseline Dice mean = 71.59% vs UnDA 82.71%; HD95 mean 21.41mm vs UnDA 13.94mm
• MM-WHS: UMMKD baseline Dice mean = 85.99% vs UnDA 82.71%; HD95 mean 23.13mm vs UnDA 13.94mm (UnDA wins HD95)
• MM-WHS Ablation: No alignment baseline Dice mean = 71.59% vs UnDA 82.71%; OT alone degrades Dice by −0.92%
• UnCT-OT alone improves Dice +10.77% over baseline; ProtoNCE alone yields inconsistent gains
• UnDA combination (UCT-OT + ProtoNCE) yields highest Dice and lowest HD95

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21546.

Fig 1: Overview of UnDA: Phase 1 trains the anchor model on source-domain im-

Fig 2 (page 3).

Limitations

• The method assumes availability of labeled anchor and target domain data, limiting applicability when target modality labels are unavailable.
• Uncertainty estimation relies on prediction entropy of the frozen anchor model, which may be suboptimal if anchor predictions are systematically biased.
• Experiments focus on two medical segmentation scenarios; generalization to other modalities, organs, or broader clinical tasks remains untested.
• No explicit adversarial or robustness evaluation against domain adversaries or out-of-distribution inputs is presented.
• Prototype memory update could be sensitive to anchor batch composition; stability under highly heterogeneous anchor data not studied.
• The method requires tuning hyperparameters for uncertainty weighting and prototype learning, and sensitivity analysis is limited.

Open questions / follow-ons

• Can UnDA be extended to fully unsupervised domain adaptation where no labeled target data is available?
• How does the uncertainty weighting perform under systematic anchor model biases or adversarial noise?
• Could the alignment approach be adapted for other cross-modal tasks beyond segmentation, such as classification or detection?
• What are the limits of modality gap size tolerable for effective alignment using this approach?

Why it matters for bot defense

Although focused on medical imaging, the UnDA framework's core principle of uncertainty-weighted alignment of semantically-structured tokens across unpaired heterogeneous data sources provides useful insights for bot-defense and CAPTCHA systems. Such mechanisms might inspire designs for robust domain adaptation or cross-input modality consistency in automated challenge-solving scenarios where paired data or perfect correspondences are not available. Uncertainty-weighted optimal transport can suppress noisy or untrustworthy supervision signals, a principle relevant in adversarial or noisy input environments typical of bot interactions. The prototype-based feature alignment technique also suggests ways to maintain stable semantic representations across varying input forms. However, direct applicability requires adaptation to the distinct characteristics of CAPTCHA data and attack modalities.

Cite

@article{arxiv2607_21546,
  title={ UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging },
  author={ Rafsan Jany and Shadab Tanjeed Ahmad and Ahsan Bulbul and Tahsinul Islam and Md Azam Hossain and Abu Raihan Mostofa Kamal },
  journal={arXiv preprint arXiv:2607.21546},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21546}
}

Read the full paper

• arXiv:2607.21546 (abstract)
• arXiv:2607.21546 (PDF)