ReMiX-MAE: Learning Missing-Channel Cross-Modal Representations from RGB-Only Clinical Facial Videos for Sympathetic-Mediated Pain Assessment

Source: arXiv:2608.02561 · Published 2026-08-03 · By Nan Bi, Taoyue Wang, Lijun Yin, Vandana Sharma

TL;DR

This paper addresses the challenge of automatically assessing sympathetic-mediated pain (SMP) from facial videos collected in real clinical settings where only RGB cameras are routinely available. Pain expression in RGB is often subtle or near-neutral, while thermal and depth modalities provide complementary physiological and geometric cues but are impractical for deployment. To bridge this modality gap, the authors propose ReMiX-MAE, a self-supervised multimodal masked autoencoder pretrained on synchronized RGB, thermal, and depth videos with explicit missing-modality simulation. This enables robust RGB-only inference by distilling cross-modal knowledge into a channel-agnostic backbone.

A novel clinical SMP dataset is collected with longitudinal, within-subject RGB videos paired with self-reported pain scores pre- and post-treatment, capturing subtle chronic pain-related facial dynamics over multiple visits. ReMiX-MAE pretrained on a large external multimodal dataset (BP4D+) demonstrates significant improvements on SMP under RGB-only deployment, outperforming RGB-only baselines and providing further gains with pseudo-multimodal reconstructed features. Cross-dataset transfer to UNBC and label-efficiency experiments on BioVid also confirm its robustness and practical utility when supervision is scarce, highlighting the advantages of cross-modal masked pretraining in clinical, data-limited pain assessment.

Key findings

• On SMP dataset, ReMiX-RGB outperforms RGB-only MAE baseline (MARLIN) on 5-class ordinal pain classification, improving subject-level accuracy from 19.0% to 23.7% and Macro-F1 from 8.2% to 9.5% (Table II).
• Incorporating pseudo-multimodal features decoded from RGB further improves 5-class accuracy to 25.5% and Macro-F1 to 11.0%, especially helping harder fine-grained separations.
• Cross-database transfer from SMP to UNBC shows ReMiX (RGB) achieves 28.0% accuracy vs 17.2% for RGB-only baseline (MAE), and ReMiX (Pseudo) highest Macro-F1 at 14.6% vs 10.3% (Table III).
• Label efficiency experiments on BioVid under 25% and 50% annotated training data demonstrate that ReMiX-Pseudo outperforms supervised TNT baseline with gains of +0.9% to +1.1% accuracy and +0.2% to +1.1% Macro-F1 (Table IV).
• Ablation removing modality-identity injection reduces SMP 5-class accuracy from 23.7% to 21.1%, and disabling episode-driven missing modality simulation reduces accuracy to 22.9%, showing both components' importance (Table V).
• SMP dataset contains 72 one-minute videos from 36 chronic SMP patients collected longitudinally pre/post treatment with video-level Numeric Rating Scale (NRS) annotations, providing richer, less skewed pain labels than existing datasets.
• Pretraining on BP4D+ multimodal facial videos without labels enables effective cross-modal representation learning transferable to clinical pain assessment from RGB-only input.

Threat model

n/a — The paper does not focus on a security or adversarial threat model. The primary challenge is modality mismatch between multimodal pretraining and RGB-only deployment in clinical pain assessment applications.

Methodology — deep read

1. Threat Model & Assumptions: The paper focuses on automated pain assessment from facial videos in real clinical settings, assuming the deployment environment provides only RGB video input due to widespread availability. Thermal and depth modalities are only available during pretraining, not at inference. The adversary is unspecified as this is not a security paper, but the model must handle missing modalities gracefully.

2. Data: The authors collected the Sympathetic-Mediated Pain (SMP) dataset with 72 one-minute RGB videos from 36 chronic SMP patients recorded longitudinally at up to four pre/post-treatment timepoints. Videos were captured with consumer smartphones under unconstrained clinical conditions. Pain labels are patient-reported Numeric Rating Scale (NRS) scores at the video level. Additional datasets used include BP4D+ (large-scale RGB, thermal, depth facial videos without pain labels) for pretraining, UNBC-McMaster Shoulder Pain dataset (clinical RGB videos with sequence-level Visual Analog Scale (VAS) pain labels), and BioVid (large-scale experimental RGB videos with stimulus-induced pain intensity).

3. Architecture & Algorithm: The core model is ReMiX-MAE, a masked autoencoder pretrained on 5-channel video inputs (RGB + depth + thermal). A novel patch-wise 5-to-3 channel routing (ROUTEMIX) dynamically selects which 3 of the 5 channels to input per patch, producing a mixed 3-channel input compatible with standard video MAE backbones. Modality-identity injection encodes which atomic channels are present per token to provide provenance information. The encoder is a video masked autoencoder that reconstructs masked patches, with separate modality-specific decoders supervising RGB patch pixel reconstruction via MSE and depth/thermal via Gaussian likelihood losses incorporating scale-invariant terms for depth. Episode-based missing modality simulation stochastically drops modalities during training to explicitly teach robustness to missing channels.

4. Training Regime: Pretraining is fully self-supervised on BP4D+, with no labels used. Each iteration samples a modality-missingness episode, routes channels patch-wise, masks 50% of tokens, encodes visible tokens, and reconstructs masked tokens across all modalities. Auxiliary losses enforce channel-drop reconstruction, temporal smoothness and frame-level consistency. Hyperparameters such as patch size, decoder widths, loss weights, and mask ratios are detailed in the supplement (not fully specified here).

5. Evaluation Protocol: Downstream evaluation occurs under RGB-only deployment on SMP, UNBC, and BioVid. The pretrained encoder is frozen; features are extracted using ReMiX-RGB (only use RGB channels) or ReMiX-Pseudo (reconstruct missing modalities from RGB and extract pseudo-multimodal features). A lightweight classifier is trained on frozen features with leave-one-subject-out (LOSO) cross-validation for SMP. Pain recognition is formulated as ordinal classification with 3, 4, or 5 pain intensity bins. Performance metrics include accuracy and macro F1 at subject and video levels. External transfer evaluates SMP-pretrained features on UNBC. Label-efficiency is measured on BioVid by training classifiers with 25% and 50% labeled data and comparing to a strong supervised Transformer-in-Transformer (TNT) baseline.

6. Reproducibility: Code release and pretrained weights were not explicitly mentioned in the excerpt, and BP4D+ is publicly available. SMP dataset is newly collected but access method is unspecified here.

Example End-to-End: To evaluate on SMP (5-class), test subjects are held out in LOSO folds. For each held-out subject, face videos are split into 5-second clips. RGB-only input clips are fed through the frozen ReMiX encoder pre-trained with missing-modality simulation. Features are extracted per clip and aggregated with MIL pooling across windows. A downstream ordinal classifier is trained on this frozen representation using sequence-level NRS labels discretized into 5 bins. Accuracy and Macro-F1 metrics are computed per subject and overall. Ablations remove modality identity injection or episode sampling to test component contributions.

Technical innovations

• Patch-wise 5-to-3 routing (ROUTEMIX) enables early fusion of 5 atomic channels (RGB, depth, thermal) into a fixed 3-channel input compatible with standard video MAE backbones, addressing modality mismatch between training and RGB-only deployment.
• Modality-identity injection encodes token-wise modality provenance via learned embeddings and a DeepSets-style aggregator, helping the encoder disambiguate heterogeneous mixed-channel tokens.
• Episode-driven missing-modality simulation stochastically drops modalities at the patch level during pretraining to explicitly train representations robust to missing input channels at deployment.
• Use of probabilistic Gaussian decoders with negative log-likelihood losses for thermal/depth reconstruction, including a scale-invariant term for depth, improves cross-modal recovery and robustness.

Datasets

• SMP (Sympathetic-Mediated Pain) — 72 one-minute RGB videos (∼110,000 frames) from 36 chronic pain patients collected longitudinally — proprietary clinical dataset
• UNBC-McMaster Shoulder Pain Archive (UNBC) — 200 short videos (∼50,000 frames) — public clinical RGB dataset with sequence-level VAS pain annotations
• BioVid — 8600 experimental RGB videos (∼1,000,000 frames) with stimulus-induced pain intensities and physiological signals — public benchmark
• BP4D+ — large-scale synchronized RGB, depth, thermal facial videos without pain labels — used for self-supervised pretraining

Baselines vs proposed

• Baseline (MAE - MARLIN RGB-only): SMP 5-class accuracy = 19.0%, Macro-F1 = 8.2% vs ReMiX-RGB: accuracy = 23.7%, Macro-F1 = 9.5% (Table II)
• Baseline (AU + PSPI handcrafted features): SMP 5-class accuracy = 18.4%, Macro-F1 = 8.2% vs ReMiX-Pseudo: accuracy = 25.5%, Macro-F1 = 11.0% (Table II)
• Baseline (MAE RGB-only): UNBC 5-class accuracy = 17.2%, Macro-F1 = 10.3% vs ReMiX-RGB: accuracy = 28.0%, Macro-F1 = 13.2%, ReMiX-Pseudo: acc = 26.8%, Macro-F1 = 14.6% (Table III)
• Supervised TNT baseline on BioVid 25% labeled data: accuracy = 25.1%, Macro-F1 = 19.9% vs ReMiX-Pseudo: accuracy = 26.0%, Macro-F1 = 20.1% (Table IV)
• Supervised TNT baseline on BioVid 50% labeled data: accuracy = 25.6%, Macro-F1 = 20.0% vs ReMiX-Pseudo: accuracy = 26.2%, Macro-F1 = 21.1% (Table IV)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02561.

Fig 1: SMP example for one subject (pre/post at initial and final visits).

Fig 2: Distribution of self-reported pain intensity in SMP (orange)

Fig 3: Overview of ReMiX-MAE (upstream pretraining). Synchronized RGB, thermal, and depth facial videos are routed from five atomic channels to a

Fig 4: ReMiX feature extraction at RGB-only deployment.

Fig 5: Representative subject-wise paired comparison on SMP under

Fig 6: and Figure 7 report the resulting subject-level

Fig 7: Subject-level test Macro-F1 under different augmentation ratios

Fig 8: Subject-wise paired comparisons on SMP under LOSO (augmentation ratio = 0.75). Left: subject-level test accuracies with the tie line. Right:

Limitations

• SMP dataset is relatively small (36 subjects, 72 videos) limiting statistical power and broader clinical generalization.
• The self-reported pain labels (NRS, VAS) are at video level only (weak labels), not frame-level or fine-grained annotations.
• No adversarial robustness or real-world deployment stress testing under varying lighting/occlusion conditions was reported.
• Pretraining dataset (BP4D+) is facial behavior but not pain-specific, so domain mismatch persists.
• Pseudo-multimodal reconstruction relies on accurate depth/thermal prediction from RGB, which may degrade under unseen clinical conditions.
• Cross-dataset evaluations focus on UNBC and BioVid only; other pain modalities, ethnicities, and illumination conditions are unexplored.

Open questions / follow-ons

• How well do ReMiX-MAE learned representations generalize to other clinical pain populations or demographics beyond SMP and UNBC?
• Can the pseudo-multimodal decoder outputs be further optimized or regularized to improve reconstruction fidelity and downstream pain performance?
• How robust is the model to extreme clinical environment conditions such as occlusions, motion blur, or low lighting?
• Would combining ReMiX-MAE with physiological or contextual cues (e.g., heart rate) further improve clinical pain assessment accuracy and robustness?

Why it matters for bot defense

The core contribution of ReMiX-MAE lies in learning robust facial video representations via self-supervised multimodal masked autoencoding with explicit missing-modality simulation. For bot-defense and CAPTCHA engineering contexts, this advances general understanding of how to learn invariant and transferable visual features when deployment-time sensing is restricted to a subset of modalities used during training. The patch-wise routing method and modality identity encoding can inspire architectures that gracefully handle partial or corrupted inputs. Additionally, the clinical pain domain highlights challenges of subtle facial cues relevant to differentiating human expressions versus synthetic or bot-generated faces. Although the pain assessment task is quite specific, the approach for cross-modal representation learning and simulating missing channels is broadly applicable to video-based behavioral analysis including bot detection and CAPTCHA challenges relying on facial motion cues. Practitioners may adapt similar masked reconstruction plus missing-modality training protocols to improve robustness and generalization of facial recognition or anti-spoofing systems under heterogeneous or degraded sensor inputs.

Cite

@article{arxiv2608_02561,
  title={ ReMiX-MAE: Learning Missing-Channel Cross-Modal Representations from RGB-Only Clinical Facial Videos for Sympathetic-Mediated Pain Assessment },
  author={ Nan Bi and Taoyue Wang and Lijun Yin and Vandana Sharma },
  journal={arXiv preprint arXiv:2608.02561},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02561}
}

Read the full paper

• arXiv:2608.02561 (abstract)
• arXiv:2608.02561 (PDF)