RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models

Source: arXiv:2608.02953 · Published 2026-08-03 · By Yuwei Ning, Liangzhi Wang, Yi Xiao, Zhenhua Wu, Yun Pang, Mingkun Chan et al.

TL;DR

RealWeather addresses the challenge of translating driving videos between clear and adverse weather conditions in a way that is both highly realistic and strictly scene-faithful. Unlike prior methods relying on synthetic paired data, 3D weather rendering, or geometry-conditioned generation, RealWeather learns authentic weather dynamics directly from unpaired real-world videos. The key innovations are Progressive Realism Bootstrapping, which iteratively refines training data by replacing crude pseudo-clear inputs with model-generated realistic videos to bridge domain gaps, and Scene-Fidelity RL Optimization, a policy optimization approach that explicitly penalizes structural hallucinations (e.g., lane changes, altered vehicle appearance) using segmentation-based rewards on safety-critical elements. RealWeather supports bidirectional weather editing (clear to adverse and adverse to clear) with unprecedented fidelity.

Extensive evaluation on standard autonomous driving datasets like Waymo Open Dataset, nuScenes, and an internal dataset show RealWeather surpasses state-of-the-art baselines in weather realism (capturing dynamic effects such as raindrops, splashes, and snow accumulation) while preserving scene geometry and object identities. It generalizes well to long-tail weather scenarios and out-of-distribution inputs including day-to-night translation and fisheye camera views. Ablation studies confirm that bootstrapping boosts realism scores from 4.47 to 4.99, and RL optimization improves scene fidelity IoU from 0.54 to 0.69. This work significantly advances realistic and structurally consistent weather video simulation for driving, critical for autonomous system validation.

Key findings

• Progressive Realism Bootstrapping raises VLM-based Realism score from 4.47 to 4.99, showing improved adaptation to real input styles.
• Scene-Fidelity RL Optimization improves SAM3 IoU for lane markings, pedestrians, and vehicles from 0.54 to 0.69, indicating better preservation of safety-critical scene elements.
• RealWeather achieves best or tied best CLIP Text-Image Similarity (CS), CLIP-Consistency (CC), PickScore (PS), and VLM-Realism (VR) metrics compared to six baselines for clear-to-adverse and adverse-to-clear translations.
• On Waymo, nuScenes, and an internal dataset, RealWeather synthesizes complex weather-scene interactions like windshield wiper effects, tire splashes, and wet-road reflections that other methods poorly capture.
• RealWeather supports robust long-tail scenario synthesis (e.g., heavy puddles with snowfall) and zero-shot generalization to day-night and fisheye camera views without retraining.
• Bootstrapping bridges the pseudo-to-real domain gap by progressively replacing pseudo-clear inputs with model-generated realistic videos during training.
• The Pseudo-Clear Generation pipeline creates temporally-coherent approximate clear videos from real adverse videos, enabling paired supervision despite lack of real paired data.
• RealWeather maintains scene structural integrity and suppresses hallucinations via reward-driven Scene-Fidelity RL Optimization using segmentation IoU rewards on lanes, pedestrians, and vehicles.

Threat model

The implicit threat model considers an adversary as the inherent domain gap and structural hallucination risks from weather translation that could mislead downstream autonomous perception and planning—an adversary corresponds to any factors causing degradation in fidelity or introducing false scene elements. The model assumes no access to paired clear/adverse videos of identical scenes, and the main capability the adversary cannot overcome is enforcement of strict scene fidelity via segmentation-based reward penalties.

Methodology — deep read

The goal is bidirectional weather translation of driving videos, i.e., converting clear weather scenes to various adverse conditions and vice versa, while preserving scene structure. The major challenges are lack of paired data and the need to ensure scene-faithfulness (avoid structural hallucinations).

1. Threat model & assumptions: The adversary is implicit—the model faces domain gaps between pseudo-clear inputs and real clear videos, and must avoid generating structural hallucinations (e.g., adding, removing, or modifying safety-critical driving elements). It is assumed that real paired videos under different weathers are unavailable at scale.

2. Data: Training uses 7,000 real-world adverse weather videos and 1,000 real clear-weather videos with 50 frames at 720p/25fps from Waymo Open Dataset, nuScenes, and a private dataset. Adverse conditions include rain, snow, and fog. There are no strictly paired clear/adverse weather video pairs.

3. Architecture & algorithm: RealWeather adapts a pretrained driving world model Cosmos-Predict2.5—a latent video diffusion model trained with flow matching. Videos are encoded into latent space by a 3D VAE and decoding is learned to reconstruct frames. The model predicts latent velocity fields conditioned on noisy latent states, timestep, input driving video embedding, and text prompt specifying weather.

To bridge input domain gaps, the model is conditioned on the encoded input video concatenated with the noisy target latent. Progressive Realism Bootstrapping iteratively replaces pseudo-clear inputs with more realistic model-generated clear videos in training pairs. Scene-Fidelity RL Optimization treats weather translation as a policy and uses segmentation IoU-based reward penalties on lanes, vehicles, and pedestrians to reduce hallucination via reinforcement learning.

4. Training regime: Training starts by paired supervision between real adverse-weather video and pseudo-clear generated by an auxiliary pipeline Ppc. Ppc converts first frame of adverse video to pseudo-clear by image editing, propagates clear style temporally via geometry-conditioned generation. The model is trained with flow matching objectives for clear→adverse and advers→clear directions, and progressively adapts to real clear inputs generated by the model in the bootstrapping loop. RL optimization periodically finetunes the model to preserve scene fidelity while maintaining weather editing capability. Specific epochs, batch sizes, and hardware details are in appendix (not fully detailed in text).

5. Evaluation: Metrics include CLIP Text-Image Similarity (weather alignment), CLIP-Consistency (temporal coherence), DINO Structure Distance (scene preservation, lower better), PickScore (learned perceptual quality), and a new VLM-based Realism score using real weather references. Baselines include simulator-supervised methods, 3D weather editing, geometry-conditioned generation, and video editing models. Ablations test effect of bootstrapping and RL optimization with quantitative gains and qualitative examples.

6. Reproducibility: The paper references pretrained models and standard datasets but the auxiliary Pseudo-Clear Generation pipeline relies on proprietary image editing models. Code and weights release status is not explicitly mentioned.

Example end-to-end: Start with a real adverse weather video xra. Apply Ppc to generate pseudo-clear xpc. During training, encode xpc and xra, add noise at timestep t, model predicts velocity field conditioned on concatenated latent zt and zpc with adverse weather prompt ca, minimizing flow matching loss. Once the model can generate realistic adverse videos from pseudo-clear inputs, it bootstraps by generating adverse videos from clear videos processed into pseudo-clear style and trains reverse direction. Then generate more realistic clear videos from real adverse inputs via the model, refining inputs in a progressive loop bridging pseudo to real. Scene-Fidelity RL Optimization fine-tunes the model by rewarding high IoU between segmentation masks of key elements in input and output videos, reducing hallucinations. The resulting model synthesizes highly realistic weather effects without compromising scene structure.

Technical innovations

• Progressive Realism Bootstrapping: An iterative training strategy that progressively replaces pseudo-clear inputs with progressively improved model-generated clear videos to bridge the pseudo-to-real domain gap in unpaired weather translation.
• Scene-Fidelity RL Optimization: A reinforcement learning approach that optimizes weather translation with segmentation IoU-based rewards on safety-critical scene elements to explicitly suppress structural hallucinations.
• Auxiliary Pseudo-Clear Generation pipeline: Constructs temporally-coherent pseudo-clear videos from single-frame image editing and geometry-conditioned propagation, enabling pseudo-paired training from unpaired real videos.
• Adapting a flow-matching latent video diffusion driving world model to perform bidirectional, video-to-video weather editing conditioned on input video latent and weather text prompt.

Datasets

• Waymo Open Dataset — ~8,000 videos (combined clear and adverse) — public
• nuScenes — subset of 300 videos (clear and adverse) — public
• Internal Driving Dataset — size not explicitly stated — private

Baselines vs proposed

• WeatherEdit: CLIP-Consistency = 0.90 vs RealWeather: 0.91 (clear-to-adverse)
• IntrinsicWeather: PickScore = 21.34 vs RealWeather: 21.52 (clear-to-adverse)
• AutoAWG: CLIP-Text Image Similarity = 0.19 vs RealWeather: 0.20 (adverse-to-clear)
• Cosmos-Transfer2.5: VLM-Realism = 2.77 vs RealWeather: 4.51 (adverse-to-clear)
• Wan2.1-VACE: DINO Structure Distance = 0.01 vs RealWeather: 0.03 (clear-to-adverse, structural preservation slightly lower but balanced with realism)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02953.

Fig 1: RealWeather enables bidirectional, realistic, and scene-faithful weather translation for driving videos. Given a sunny

Fig 2: Overview of RealWeather. (A) Progressive Realism Bootstrapping enables the model to learn weather effects directly

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The auxiliary Pseudo-Clear Generation pipeline depends on pretrained image editing and geometry-conditioned models, potentially limiting reproducibility and generality.
• Quantitative evaluation is limited to standard metrics; no extensive adversarial attacks or real downstream autonomous driving system validation reported.
• The datasets, while covering common weather types, may still be limited in scale or diversity compared to all real driving environments.
• Some metrics (e.g., DINO Structure Distance) show RealWeather slightly lower structural preservation than certain conservative baselines, indicating a tradeoff between realism and preservation.
• Details on training hyperparameters, compute requirements, and random seeds are not reported, limiting precise reproducibility.

Open questions / follow-ons

• How does RealWeather perform under extreme out-of-distribution scenarios such as heavy fog or hail which were not explicitly tested?
• What is the impact of weather translation errors on downstream perception and planning models in autonomous driving stacks?
• Can the Progressive Realism Bootstrapping strategy be generalized to other video-to-video conditional generation tasks beyond weather?
• How to further improve structural preservation without sacrificing realistic weather effect synthesis?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, RealWeather demonstrates an advanced approach to generating highly realistic, dynamic visual transformations that strictly preserve key scene structures. This methodology underscores the importance of maintaining fidelity to underlying content while applying complex visual edits—key to ensuring that synthesized training data remain valid for security or robustness evaluations. The combination of iterative training bootstrapping to reduce domain gaps and reinforcement learning with targeted rewards to prevent hallucinations could inspire analogous approaches in CAPTCHA generation or verification systems where adversarial robustness and visual realism must be balanced. Moreover, the handling of unpaired data via pseudo-counterpart generation may inform data augmentation pipelines in security applications where collecting paired data is infeasible.

Cite

@article{arxiv2608_02953,
  title={ RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models },
  author={ Yuwei Ning and Liangzhi Wang and Yi Xiao and Zhenhua Wu and Yun Pang and Mingkun Chan and Jichang Li and Guanbin Li },
  journal={arXiv preprint arXiv:2608.02953},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02953}
}

Read the full paper

• arXiv:2608.02953 (abstract)
• arXiv:2608.02953 (PDF)