Spatio-Temporal Conditional Denoising Transformer for Modality-Missing RGBT Tracking

Source: arXiv:2607.24701 · Published 2026-07-27 · By Andong Lu, Ziyi Zha, Jiandong Jin, Shihao Li, Chenglong Li, Jin Tang et al.

TL;DR

This paper addresses the challenge of modality-missing in RGB-Thermal (RGBT) video object tracking, where sensor failures or occlusions cause one modality (RGB or thermal) to be missing intermittently during tracking. Existing approaches rely heavily on spatial cues from current frames and require separate model branches or expert modules to handle missing and complete modalities, leading to inflexibility and temporally inconsistent reconstructions. The authors propose a unified framework called the Spatio-Temporal Conditional Denoising Transformer (SCDT), which formulates multimodal feature reconstruction and enhancement as a denoising process conditioned jointly on spatial cues from the current frame and temporal contexts extracted both from recent short-term and long-term historical frames across modalities. By modulating the noise level injected into inputs, SCDT adapts its reconstruction or enhancement objectives dynamically without changing model parameters or architecture. Extensive experiments across three large-scale public RGBT tracking benchmarks with modality-missing variants demonstrate that SCDT outperforms state-of-the-art methods consistently in both modality-complete and modality-missing settings, achieving precision and success rates gains of up to 4.2% and establishing a new robust baseline for modality-missing RGBT tracking.

Key findings

• SCDT achieves new state-of-the-art precision rates on modality-missing benchmarks: 69.3% PR on LasHeR-Miss, beating FlexTrack by 4.2%.
• On RGBT234-Miss, SCDT sets new best mean precision rate (MPR) of 88.1% and mean success rate (MSR) of 64.3%, significantly surpassing recent ViT-based trackers.
• Dual temporal conditioning combining short-term and long-term temporal cues improves performance by approximately +1.4% PR and +1.3% SR on LasHeR and +1.0%/+0.8% on LasHeR-Miss compared to single conditioning strategies.
• Noise-modulated adaptation with weak noise for complete modality and strong noise for missing modality scenarios yields better tracking performance (+1.6% PR on LasHeR and +0.9% PR on LasHeR-Miss) than applying uniform noise levels.
• Combining reconstruction loss for missing modalities and statistical alignment loss for complete modalities outperforms either loss alone, benefiting stable feature generation.
• A moderate denoising transformer depth of 4 layers is optimal; shallower (2 layers) reduces performance and deeper (6 layers) causes slight over-smoothing.
• Qualitative visualizations show enhanced feature maps concentrate sharply on targets and reconstructed missing modality features closely match ground truth both spatially and semantically.
• SCDT demonstrates robust performance across diverse attribute challenges including occlusion, illumination variation, motion blur, and high missing-ratio settings (>90% modality missing).

Threat model

The adversary is assumed to be environmental or hardware failures causing intermittent or prolonged missing modalities (either RGB or thermal) during tracking, such as occlusion, sensor malfunction, or misalignment. The system cannot rely on access to the missing modality at certain frames but has access to past frames from both modalities and the currently available modality. The adversary cannot arbitrarily manipulate both modalities simultaneously nor inject adversarial perturbations beyond causing modality absence.

Methodology — deep read

The threat model assumes an RGB-Thermal tracking scenario where one modality can be intermittently missing due to sensor failure, occlusion, or misalignment. The adversary is not explicitly defined, but robustness under real-world operational modality degradations is targeted.

Data provenance includes three public large-scale datasets for RGBT tracking: LasHeR (~1.47M frames), RGBT234 (~116k frames), and VTUAV (~1.7M frames), with modality-missing variants generated by simulating missing frames according to patterns such as long-time missing or random missing.

The overall architecture uses a ViT-Base (ViT-B) backbone pretrained from the ODTrack model for unimodal encoding of RGB and thermal inputs independently. Extracted features from template and search frames are perturbed by additive Gaussian noise with noise variance adapting to task (strong noise for missing reconstruction, weak noise for enhancement).

The core is the Spatio-Temporal Conditional Denoising Transformer (SCDT) module implemented as a transformer-based denoiser that fuses noisy features with spatio-temporal condition tokens. Two temporal conditioning schemes are used: (1) short-term cross-attention to fuse noisy input features with spatial cues from recent neighboring frames of the complementary modality, capturing fine-grained motion continuity; (2) long-term FiLM-style modulation using learned tokens encoding global temporal context to enforce semantic consistency and suppress noise.

The module takes noisy feature tokens from the available modality as input along with short-term spatial tokens and a single long-term modality token, producing refined features that reconstruct missing modalities or enhance weak ones.

Training involves dynamically adjusting losses and noise levels according to modality availability. For missing modalities, a mean square error reconstruction loss supervises the denoised output against the ground-truth feature of the missing modality. For complete modalities, a statistical alignment loss matches the mean and variance of the refined features to the original ones, encouraging discriminative enhancement.

Losses are combined with a standard tracking loss (bounding box regression) from ODTrack, with weighting factors switching to prioritize relevant objectives per modality condition.

The model is trained on 6 NVIDIA RTX 4090 GPUs using AdamW optimizer at batch size 24. Learning rates are set to 1e-5 for backbone and 1e-4 elsewhere, weight decay starts after epoch 24. Training epochs are 30 for LasHeR and RGBT234 variants and 5 for VTUAV variants. Each epoch contains 40k-60k samples.

Evaluation follows the standard one-pass evaluation (OPE) protocol measuring precision rate (PR) and success rate (SR) on both complete and modality-missing benchmarks. Ablations dissect the contributions of temporal conditions, noise strategies, loss components, and denoiser depth. Qualitative visualizations illustrate feature maps under different modality availabilities.

Reproducibility: The model builds on publicly known ViT-B and ODTrack pretrained weights. Code is publicly released as per the paper. Datasets used are publicly available, with modality-missing variants constructed by the authors.

An end-to-end example: Given a sequence with missing thermal frames, RGB features are extracted by the ViT-B encoder, Gaussian noise with high variance is applied to simulate corruption, and the SCDT denoiser conditioned on short-term RGB and long-term temporal tokens progressively reconstructs thermal features. These reconstructed features are concatenated with the noisy RGB features and passed through the tracking head to predict bounding boxes. The entire model is trained to minimize reconstruction error on thermal features along with the tracking loss to ensure accurate target localization despite missing modalities.

Technical innovations

• Unified spatio-temporal conditional denoising transformer framework that handles both modality reconstruction and feature enhancement without changing architecture or parameters.
• Dual temporal conditioning mechanism combining cross-attention with short-term temporal tokens and FiLM-based long-term temporal token modulation for robust multimodal feature refinement.
• Noise-modulated adaptation strategy that adjusts noise intensity to dynamically switch between reconstruction (strong noise) and enhancement (weak noise) tasks within the same model.
• Task-specific dual supervision combining feature-level reconstruction loss for missing modalities and statistical alignment loss for complete modalities.
• Moderate depth transformer denoiser design optimized for balancing feature denoising capacity and maintaining temporal feature stability.

Datasets

• LasHeR — 1,224 sequences, 1.47 million frames — public RGBT tracking dataset
• LasHeR-Miss — 245 sequences, 220.7K frames with 133.2K missing frames — modality-missing variant constructed by authors
• RGBT234 — 234 sequences, 116.7K frames — public RGBT tracking dataset
• RGBT234-Miss — 234 sequences, 116.7K frames with 69.2K missing frames — modality-missing variant constructed by authors
• VTUAV — 500 sequences, 1.7 million frames — public UAV-based RGBT tracking dataset
• VTUAV-Miss — 176 sequences, 631.5K frames with 368.5K missing frames — modality-missing variant constructed by authors

Baselines vs proposed

• FlexTrack (ICCV 2025): LasHeR-Miss PR = 65.1% vs SCDT = 69.3%
• FlexTrack: RGBT234-Miss MPR = 84.1% vs SCDT = 88.1%
• IPL (IJCV 2025): LasHeR PR = 69.4%, SR = 55.3% vs SCDT PR = 77.4%, SR = 61.0%
• STTrack (AAAI 2025): LasHeR PR = 76.0%, SR = 60.3% vs SCDT PR = 77.4%, SR = 61.0%
• AINet (AAAI 2025): VTUAV PR = 88.0%, SR = 75.3% vs SCDT PR = 93.6%, SR = 78.9%
• Ablation on temporal conditioning: addition of short-term and long-term conditions improves LasHeR PR from 75.1% baseline to 77.4% (full model)
• Ablation on noise modulation: weak-strong noise strategy yields LasHeR PR 77.4% vs uniform strong noise 73.2%
• Ablation on denoiser depth: 4 layers achieve 77.4% PR vs 2 layers 75.9% and 6 layers 76.4%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24701.

Fig 1: Comparison of the existing method of RGBT tracking

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 2: Overall architecture of the proposed SCDT. SCDT performs a noising-denoising process on available search frames, guided

Fig 3: Detailed architecture of the Spatio-temporal Conditioned

Fig 4: Visualization of features learned by SCDT under com-

Fig 5: Precision rate (PR) of challenge attributes on LasHeR-Miss dataset. The axes of each attribute have been normalized.

Limitations

• The paper evaluates reconstruction quality and tracking robustness, but does not provide detailed adversarial robustness analysis under deliberate attacks targeting modality failures.
• Temporal conditioning relies on historical frames being available and consistent; abrupt or prolonged modality outages beyond training distributions may degrade performance.
• The modality-missing variants are synthetically constructed; real-world missing modality noise characteristics or correlated sensor failures are not explicitly modeled or analyzed.
• Ablation studies focus on performance metrics but lack detailed runtime or computational overhead analysis critical for real-time tracking deployment.
• The four-layer denoising transformer design is fixed; further exploration of adaptive or dynamic depth mechanisms is not investigated.
• Cross-dataset generalization and domain adaptation capabilities under various environmental factors remain unexplored.

Open questions / follow-ons

• How does SCDT perform under real-world noisy or corrupted modality inputs beyond complete missing frames, such as low-quality or corrupted sensor data?
• Can the framework be extended to handle more than two modalities or more complex modality missing patterns in multimodal tracking beyond RGB-T?
• What is the computational trade-off and latency impact of the spatio-temporal conditional denoising framework for real-time tracking applications?
• How robust is the method against intentional adversarial attacks targeting missing modality exploitation or temporal inconsistency?

Why it matters for bot defense

Bot-defense systems and CAPTCHA challenges often rely on multimodal signals or temporal consistency to detect automated attacks. This work introduces a principled conditional denoising approach that dynamically handles missing or corrupted modality inputs in a unified framework leveraging spatio-temporal context, which can inspire similar defense mechanisms when some sensor or signal channels become unavailable or noisy. The noise-modulated adaptation also suggests ways to flexibly switch between reconstruction and enhancement objectives to maintain robust feature integrity under partial observation. The dual temporal conditioning that leverages both local short-term and global long-term context could improve stability of multimodal representations over time in security-critical vision systems. However, direct application to CAPTCHAs would require adaptation to their specific modalities and operational constraints. Overall, this research provides a novel strategy to robustly fuse incomplete multimodal data streams which is highly relevant for designing resilient visual bot-detection and challenge-response pipelines that experience sensor degradation or partial observability.

Cite

@article{arxiv2607_24701,
  title={ Spatio-Temporal Conditional Denoising Transformer for Modality-Missing RGBT Tracking },
  author={ Andong Lu and Ziyi Zha and Jiandong Jin and Shihao Li and Chenglong Li and Jin Tang and Bin Luo },
  journal={arXiv preprint arXiv:2607.24701},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24701}
}

Read the full paper

• arXiv:2607.24701 (abstract)
• arXiv:2607.24701 (PDF)