Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification

Source: arXiv:2607.24683 · Published 2026-07-27 · By Francisco Mena, Dino Ienco, Roberto Interdonato, Cassio F. Dantas, Simon Besnard

TL;DR

This paper tackles the challenge of multi-modal classification when an arbitrary subset of input modalities may be missing at inference time—a common scenario in real-world applications due to sensor failures, privacy restrictions, or operational constraints. Unlike prior work mainly focused on bimodal setups or robust fusion for predefined missing patterns, the authors propose a multi-modal co-learning framework emphasizing inter-modal collaboration via feature- and decision-level knowledge sharing. They introduce two novel methods: Co-Miss, which applies missing modality distillation with exhaustive missing modality simulation, and FullCo, which uses mutual distillation across all modalities to guide unimodal predictions toward full-modality consensus. Evaluations on two multi-modal benchmarks with 4 and 19 modalities respectively demonstrate that Co-Miss excels under minimal missing conditions (single missing modality), while FullCo is more robust under moderate and extreme missing cases (many or all-but-one missing). Both methods outperform a suite of recent baselines across missing modality scenarios, improving classification robustness significantly. Ablations reveal the importance of volume-based contrastive learning for shared representation and distillation losses for decision-level collaboration. The paper provides a comprehensive solution to the missing arbitrary modalities problem with practical insights for diverse multi-modal domains.

Key findings

• Co-Miss reduces performance drop under minimal missing conditions by up to 6.6 F1 points compared to the Dec baseline, e.g., dropping 12.1 vs 18.7 points on missing optical modality (Table 1).
• FullCo achieves highest F1 scores under moderate and extreme missing modality conditions, outperforming all baselines by 3-6 absolute F1 points (Table 1).
• Both methods tie for best average rank across all missing cases with an average rank of 1.5 on Multi-CropHarvest and rank 2.4 for FullCo on HL-Opportunity (Tables 1 and 2).
• Volume-based multi-modal contrastive loss yields 1-2 F1 points better robustness than standard pairwise contrastive loss (Table 3).
• Ablations show removing decision-level distillation losses causes largest performance drop under extreme missing, while removing feature-level losses hurts minimal missing cases most (Fig 5).
• Performance degrades non-linearly as more modalities go missing; Co-Miss and FullCo maintain higher F1 and lower variance than competitors as missing modalities increase (Fig 4).
• FullCo is recommended for >10 modal scenarios due to exponential training cost of Co-Miss’s exhaustive missing simulation (Section 4.4).
• On the large HL-Opportunity dataset with 19 sensor modalities, performance drop is smaller for minimal missing due to sensor redundancy; FullCo almost matches best baselines (Table 2).

Threat model

The adversary represents operational or environmental constraints causing arbitrary missing subsets of modalities at inference time, without predefined or predictable patterns. The adversary cannot interfere with model training or modify inputs but can restrict availability of any combination of modalities at deployment, forcing the model to maintain robust predictions despite incomplete modal input.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is the environment or operational constraints causing arbitrary missingness—any subset of modalities may be unavailable at inference time without prior pattern knowledge. Models are trained on full-modality data and must adapt to arbitrary missing subsets later. The adversary does not disrupt training but can control modality availability at test time.

2. Data: Two benchmarks are used. Multi-CropHarvest consists of 29,642 samples with four modalities (Sentinel-2 optical, Sentinel-1 radar, weather, topographic features) for crop-type classification with 10 classes; evaluated via 10-fold cross-validation. HL-Opportunity has 19 wearable sensor modalities across 6 subjects in 5 activity classes with about 264k train, 6.5k validation, and 12.6k test samples. Data preprocessing includes standard z-score normalization, and class imbalance is handled by weighted loss.

3. Architecture / Algorithm: Each modality is encoded separately using modality-dedicated encoders (1D CNNs for multi-temporal, MLPs for mono-temporal), producing two latent representations per modality: shared features zsha_m and specific features zspe_m. Shared features capture modality-invariant, class-relevant aspects; specific features capture unique modality information.

   Per-modality predictions are obtained by concatenating zsha_m and zspe_m followed by a linear classification head.

   The main fusion produces a consensus prediction by averaging per-modality softmax probabilities across all available modalities.

   Co-learning includes:

   • A modality discriminant classifier Lmod that enforces zspe_m features to identify their modality.
   • A volume-based contrastive loss Lcont on zsha_m features to enforce proximity of shared features across modalities for same samples.

4. Decision-Level Collaboration / Knowledge Distillation:

   • Co-Miss: Employs missing distillation loss Lmiss by simulating all possible missing modality subsets exhaustively (2^M - 1 combinations). Each missing combination's fused prediction ˆy_miss is enforced to match ground truth and full-modality consensus through cross-entropy and softened KL divergence (knowledge distillation) losses. ModDrop is used for stochastic masking during these simulations.

   • FullCo: Uses mutual distillation loss Lmut where each modality’s individual prediction is guided to imitate the full-modality consensus and ground truth, focusing on unimodal robustness especially under extreme missing conditions.

5. Training Regime:

   Optimized via AdamW optimizer with 10^-3 learning rate, batch size 128, early stopping based on macro F1 on validation. Two-layer encoders (128 units) for Multi-CropHarvest and four-layer encoders (64 units) for HL-Opportunity with 20% dropout. Loss terms Lmain, Lmod, Lcont, and either Lmiss (Co-Miss) or Lmut (FullCo) are combined unweighted.

6. Evaluation Protocol:

   Weighted F1 scores computed for full modality and all missing modality scenarios grouped into minimal (single missing), moderate, and extreme (all-but-one present). Macro missing score averages across these modes. Statistical significance tested via Welch's t-test.

7. Reproducibility:

   Code is publicly released at https://github.com/fmenat/Co4Miss with full training and evaluation scripts. Datasets are open or publicly referenced.

Concrete Example: For M=4 modalities (Multi-CropHarvest), Co-Miss training simulates all 15 possible missing modality subsets. For each training sample, encoders generate feature embeddings zsha and zspe per modality, classify modality via Lmod, align shared features with Lcont, produce per-modality predictions, and aggregate full-modality consensus. Missing modality combinations’ aggregated predictions are computed by ignoring masked modalities; these are taught to mimic full-modality consensus through cross-entropy and distillation loss. This forces unimodal and partial-modal models to collaborate and agree, enhancing robustness when modalities go missing. Evaluation at test time averages per-modality predictions for whatever subset is available without any explicit imputation or synthesis.

Technical innovations

• Introduction of a multi-modal co-learning framework prioritizing inter-modal collaboration over fusion to handle arbitrary missing modalities flexibly.
• Development of two complementary decision-level knowledge distillation strategies: missing distillation with exhaustive masking (Co-Miss) and mutual distillation guiding unimodal predictions (FullCo).
• Use of volume-based multi-modal contrastive loss to structure modality-shared feature spaces more effectively than standard pairwise contrastive methods.
• Exhaustive simulation of all missing modality subsets during training to enforce robustness across the full spectrum of missing conditions, unlike prior bimodal or partial missingness work.

Datasets

• Multi-CropHarvest — 29,642 samples — public dataset for crop-type classification from Sentinel optical, radar, weather, and topographic data
• HL-Opportunity — 283,495 samples (train+val+test) — public human activity recognition dataset with 19 wearable sensor modalities

Baselines vs proposed

• Dec baseline with zero-imputation: minimal missing F1 drops up to 17.1 points vs Co-Miss reducing drop to 12.1 points on optical modality missing (Multi-CropHarvest)
• Co-Miss: macro missing F1 = 70.1 vs DSensD+ (best competitor) = 65.8 (Multi-CropHarvest)
• FullCo: macro missing F1 = 70.4 vs DSensD+ = 65.8 (Multi-CropHarvest)
• FullCo ranks 2nd in HL-Opportunity with macro missing F1 = 72.0 vs DSensD+ = 72.7 (difference not statistically significant)
• Volume-based contrastive loss surpasses pair-wise contrastive loss by 1-2 F1 points in Multi-CropHarvest robustness
• Removing decision-level distillation losses drops F1 by up to 2 points in extreme missing cases; removing feature-level losses drops up to 3 points in minimal missing (Fig 5)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24683.

Fig 4: F1 score by increasing the number of modalities missing at inference time.

Fig 5: F1 gap between the default version of our methods and different variants.

Fig 3 (page 13).

Limitations

• Training complexity of Co-Miss grows exponentially (O(2^M)) with number of modalities, becoming computationally impractical beyond ~10 modalities.
• Evaluations mainly focus on sensor-based multi-modal benchmarks; generalization to mainstream image-text-audio multi-modal tasks is not shown.
• No explicit adversarial evaluation of worst-case missing modality patterns or compositionally novel missing modality subsets at inference beyond random masking.
• Model architectures follow standard encoders without exploration of advanced modality-specific backbone designs or attention mechanisms.
• The modality discriminant loss Lmod contributes least to performance, indicating potential underutilization of modality-specific features.
• Robustness under severe missing modality conditions depends heavily on decision-level distillation whose effectiveness may vary with modality redundancy.

Open questions / follow-ons

• How can the Co-Miss training complexity be reduced or approximated to scale efficiently to larger modality counts without exhaustive enumeration?
• Can adaptive mechanisms dynamically balance modality-shared vs. modality-specific information depending on input modality availability?
• How do these co-learning methods generalize to mainstream multi-modal tasks with heterogeneous modalities like image-text-audio combinations?
• What are the theoretical limits of decision-level distillation in enabling unimodal robustness when many modalities are missing?

Why it matters for bot defense

Bot-defense systems often rely on multi-modal signals such as behavior patterns, device fingerprints, mouse dynamics, and network metadata. These modalities may not always be available due to privacy constraints, user agent variability, or sensor failures—mirroring the missing arbitrary modality scenario studied here. The proposed co-learning framework offers a principled way to train models that remain robust when any subset of modalities is missing, by fostering inter-modal collaboration and knowledge distillation rather than relying solely on fusion.

Specifically, a CAPTCHA or bot-detection engineer could apply the co-learning paradigm to combine heterogeneous behavioral signals while tolerating missing inputs at inference time without degrading detection performance drastically. The distinction between Co-Miss (better for scenarios where a small number of signals are missing) and FullCo (better when most signals are missing) helps adapt defenses dynamically depending on expected modality loss patterns. Overall, the methodology advances the state of the art for multi-modal robustness in adversarial and constrained deployment environments relevant to real-world bot-defense applications.

Cite

@article{arxiv2607_24683,
  title={ Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification },
  author={ Francisco Mena and Dino Ienco and Roberto Interdonato and Cassio F. Dantas and Simon Besnard },
  journal={arXiv preprint arXiv:2607.24683},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24683}
}

Read the full paper

• arXiv:2607.24683 (abstract)
• arXiv:2607.24683 (PDF)