Boosting Robustness for All-Weather Self-Supervised Depth Estimation in Autonomous Driving

Source: arXiv:2607.21526 · Published 2026-07-23 · By Mengshi Qi, Xiaoyang Bi, Xianlin Zhang, Huadong Ma

TL;DR

This paper addresses the problem of robust self-supervised monocular depth estimation for autonomous driving under diverse and adverse weather conditions (night, rain, fog, snow). Conventional methods degrade significantly in these conditions because weather phenomena violate core assumptions such as brightness constancy, causing erroneous pixel correspondences and poor depth predictions. Moreover, while radar is a valuable sensor in adverse weather due to its robustness, its sparse data distribution in the camera's Point of View (POV) limits effective fusion. The authors propose a novel training pipeline combining Uncertainty-Aware Multi-Teacher Distillation (UAMTD) and a new POV-BEV Radar Fusion strategy that leverages denser Bird's-Eye View (BEV) radar data for improved fusion with camera features.

UAMTD generates diverse “weather-expert” teacher models specialized on different weather subsets, and uses an uncertainty estimation branch to weigh the contribution of each teacher’s pseudo-label regions during distillation to a student model trained on all-weather data. The POV-BEV Radar Fusion exploits camera-pixel ray constraints to cross-attend dense BEV radar features back into the camera's POV feature space, complementing sparse POV radar points. Experiments on RADIATE and nuScenes all-weather driving datasets show 20-85% relative absRel error reduction over direct all-weather training baselines and consistent improvements over prior state-of-the-art robust depth methods. The approach also generalizes across several architecture backbones, demonstrating broad applicability.

The key contributions are a novel uncertainty-aware multi-teacher distillation approach allowing selective knowledge transfer from multiple adversarial weather expert models and a principled cross-view camera-radar fusion method exploiting radar point density in BEV. These innovations enable robust, self-supervised monocular depth estimation suitable for safety-critical autonomous driving across challenging real-world weather conditions.

Key findings

• UAMTD applied to Lite-Mono reduces absRel error by 20%, 85%, 15%, 34%, and 28% across clear, night, rain, fog, and snow conditions respectively compared to direct all-weather training.
• On the RADIATE dataset, the proposed method achieves a 26% absRel error reduction compared to state-of-the-art methods.
• On nuScenes, the method reduces night condition absRel error by 23% over previous state-of-the-art.
• BEV radar provides on average 239× more non-zero points per frame than POV radar (98,840 vs. 413), enabling denser geometric cues for fusion.
• The uncertainty-aware distillation loss dynamically weighs contributions from multiple teachers based on pixel-wise uncertainty, improving robustness in corrupted regions.
• Applying UAMTD to multi-frame ManyDepth2 architecture stabilizes training that otherwise deteriorates under direct all-weather training, e.g. absRel rises from 0.150 to 0.208 at night without UAMTD.
• POV-BEV radar fusion with cross-attention improves robustness over baselines fusing radar only in POV.
• The uncertainty estimation branch jointly compares multiple teachers and the student model, enabling selective knowledge transfer based on relative confidence.

Threat model

The threat model focuses on environmental perturbations caused by adverse weather conditions (rain, fog, snow, low illumination at night) that degrade sensor inputs and violate self-supervised learning assumptions such as brightness constancy and static scenes. There is no consideration of adversarial or malicious actors manipulating sensor data. The method aims to robustly estimate depth despite natural environmental challenges.

Methodology — deep read

The paper addresses robust self-supervised monocular depth estimation under adverse weather through two key innovations: Uncertainty-Aware Multi-Teacher Distillation (UAMTD) and POV-BEV Radar Fusion.

1. Threat Model & Assumptions: The adversary is environmental adverse weather and illumination variation causing violations of brightness consistency assumptions used in self-supervised depth loss. No active adversary with malicious intent. The goal is to achieve robust depth estimation despite these conditions.

2. Data: The method uses unpaired real all-weather driving data from RADIATE and nuScenes. RADIATE has 63,359 training and 10,679 validation frames across clear, night, rain, fog, and snow, though snow is absent in training. nuScenes training uses 15,129 synchronized samples across clear, night, and rain; validation has 6,019 samples divided similarly.

3. Architecture/Algorithm: The pipeline begins with generating diverse teacher models specialized in distinct weather subsets (clear, rain, fog, night), trained via Single-Teacher Distillation (STD). Each teacher is trained on its weather subset plus clear data to produce robust pseudo-labels.

The student model learns from all-weather data via a combined loss: photometric reprojection loss (Lp) plus similarity loss (Lsim) to pseudo-labels. Lsim alone risks overfitting to teacher assumptions; Lp allows adaptation.

An Uncertainty Estimation Branch (UEB) jointly assesses multiple teacher pseudo-labels and student features to estimate pixel-wise uncertainty maps at multiple stages. This uncertainty weights the multi-teacher distillation loss, down-weighting unreliable teacher predictions.

POV-BEV Radar Fusion encodes radar data from POV and BEV with lightweight ResNet-18 backbones. BEV radar offers 239× denser points than POV. A cross-attention mechanism projects BEV radar features back to the camera’s POV pixel rays, enabling dense complementary fusion. The fused radar features are concatenated with camera features and further refined.

4. Training Regime: Teachers are trained with STD loss fixed after training. The student model and UEB are trained end-to-end with uncertainty-aware multi-teacher distillation loss (Lud). Training uses AdamW optimizer with learning rate 1e-4, weight decay 1e-2, batch size 12 on a single NVIDIA 3090 GPU. Data augmentations (brightness, saturation, hue jitter, flips) applied.

Training alternates between camera-only and camera-radar paired inputs due to sensor FPS mismatch. Training epochs until convergence take ~13.5-15 hours on nuScenes.

5. Evaluation Protocol: Performance measured on held-out validation sets of two real-world all-weather datasets, using error metrics absRel, RMSE, and accuracy threshold δ<1.25, capped at 80m depth. Comparisons include camera-only baselines trained on clear or all-weather data, prior state-of-the-art robustness methods using synthetically augmented data, and camera-radar fusion methods adapted to their setting.

Ablations show gains from individual components: multi-teacher vs single teacher, uncertainty weighting vs equal weighting, and fusion of BEV radar vs POV alone. Stability tested across different backbone architectures including single- and multi-frame models.

6. Reproducibility: Code and trained models are released at the provided GitHub URL. The two datasets are public benchmarks. The method uses standard deep learning modules. Some minor details (exact random seeds or data splits) are not explicitly specified but can be inferred from standard dataset protocols.

Example: To mitigate night condition failure, a night-specialized teacher produces pseudo-labels with higher confidence in well-lit areas but high uncertainty around flickering headlights. The student model’s uncertainty-aware loss downweights reliance on unreliable night teacher pixels while adapting through photometric loss where valid, resulting in more accurate depth in night scenes.

Technical innovations

• Uncertainty-Aware Multi-Teacher Distillation enabling selective pixel-wise knowledge transfer from multiple specialized weather-expert teachers by jointly estimating inter-teacher uncertainty.
• POV-BEV Radar Fusion using camera-pixel ray constraints to perform cross-attention that projects dense BEV radar features back into camera POV feature space for more effective multi-modal fusion.
• Diverse Teacher Generation strategy that trains multiple teachers on disjoint weather subsets plus clear data, fostering specialization while anchoring to clear-weather consistency.
• Coarse-to-fine multi-stage uncertainty estimation branch that integrates student features with teacher pseudo-labels to produce refined uncertainty maps guiding selective distillation.

Datasets

• RADIATE — 63,359 training + 10,679 validation frames — real-world all-weather driving data covering clear, night, rain, fog, snow (no snow in train)
• nuScenes — 15,129 training + 6,019 validation images — urban autonomous driving with day-clear, night, and rain conditions

Baselines vs proposed

• MonoDepth2 clear training: absRel = 0.1822 night vs UAMTD all-weather: absRel = 0.1647 (10% improvement)
• Lite-Mono direct all-weather training: absRel night = 0.3059 vs UAMTD all-weather: 0.1647 (46% improvement)
• ManyDepth2 direct all-weather training night absRel = 0.208 vs UAMTD all-weather night absRel = 0.164 (21% improvement)
• RNW method (clear+night) absRel night = 0.220 vs UAMTD all-weather night absRel = 0.1647
• Camera-radar fusion method R4Dyn: absRel = 0.213 vs UAMTD+PBCRF fusion: 0.154 (28% improvement)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21526.

Fig 1: (a) The state-of-the-art self-supervised depth estimation model, Lite-

Fig 2: presents an overview of our proposed method. Our

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The approach assumes availability of diverse labeled or filtered weather subsets to train specialized teachers; may be limited where such data is not readily available.
• No explicit adversarial or malicious attack evaluation; robustness evaluated only under natural weather domain shift.
• While radar fusion improves performance, reliance on specific radar sensor types and configurations (e.g., availability of BEV radar) may limit generalization to other sensor suites.
• The uncertainty estimation branch is learned without explicit supervision, relying on indirect consistency signals; may be less effective if teachers are all poor on some conditions.
• No detailed evaluation on computational overhead or real-time latency impact of the proposed fusion and distillation modules.

Open questions / follow-ons

• Can uncertainty-aware multi-teacher distillation be extended to explicitly model temporal consistency or moving object semantics for further robustness?
• How does the method perform under combination or rapid transitions of multiple adverse weather types (e.g., night rain or fog with snow)?
• Would fusing additional sensor modalities (e.g., LiDAR, thermal) benefit from similar uncertainty-aware cross-modal distillation frameworks?
• Can the uncertainty estimation branch be improved with explicit uncertainty supervision or adversarial robustness objectives?

Why it matters for bot defense

Though this work focuses on robust depth estimation for autonomous driving, the principles of uncertainty-aware multi-teacher distillation and cross-modal fusion under challenging sensory conditions are highly relevant to CAPTCHA and bot-defense systems. CAPTCHA challenges often face noisy or obfuscated inputs similar to adverse weather effects here, and multi-expert distillation with uncertainty weighting could improve robustness against noisy or adversarial image inputs. Additionally, the POV-BEV cross-attention fusion concept shows how exploiting complementary sensor perspectives can enhance robust feature representation. The ideas of explicitly modeling uncertainty to selectively weight training signals could help CAPTCHA systems better learn from ambiguous or noisy challenge-response pairs, reducing failure rates under real-world distortions or attacks. However, direct application requires adapting the input modalities and threat models. Researchers designing CAPTCHA solvers or bot detectors should consider multi-expert uncertainty-aware distillation as a technique to improve model reliability and generalization in adverse or adversarial input conditions.

Cite

@article{arxiv2607_21526,
  title={ Boosting Robustness for All-Weather Self-Supervised Depth Estimation in Autonomous Driving },
  author={ Mengshi Qi and Xiaoyang Bi and Xianlin Zhang and Huadong Ma },
  journal={arXiv preprint arXiv:2607.21526},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21526}
}

Read the full paper

• arXiv:2607.21526 (abstract)
• arXiv:2607.21526 (PDF)