DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning

Source: arXiv:2607.16090 · Published 2026-07-17 · By Hanyang Chen, Anirudh Satheesh, Longchao Da, Hua Wei

TL;DR

This paper addresses the challenge of transferring reinforcement learning (RL) policies from a source domain (e.g., a simulator) to a target domain with differing dynamics, a common problem in sim-to-real adaptation. Unlike prior approaches that use domain classifiers, value-guided filtering, or latent representation learning, the authors propose a novel approach based on generative diffusion models to measure and adapt to dynamics mismatch. Their key insight is to view state transitions as a generative process with latent diffusion states forming a generative trajectory, and that the divergence between trajectories from source and target domains quantifies dynamics mismatch more finely than prior methods. They introduce DADiff, a diffusion-driven method that estimates deviation using the latent noise predictions of diffusion models trained on source and limited target domain data. They propose two adaptation variants: DADiff-modify adjusts source rewards with a penalty based on divergence, while DADiff-select filters source data by deviation before value-function updates. The authors prove a theoretical performance bound relating policy performance difference across domains to the generative trajectory deviation. Extensive experiments on MuJoCo environments with various kinematic, morphology, friction, and gravity shifts demonstrate that DADiff variants consistently outperform multiple baselines, showing both strong and stable adaptation despite limited target domain samples. A detailed parameter study and analysis explain the complementary strengths of the reward modification versus data selection variants under different shift types and stochasticity. Overall, this work contributes theoretically grounded, practically effective diffusion-based dynamics adaptation approaches that outshine existing state-of-the-art methods in cross-domain RL policy transfer.

Key findings

• Theorem 4.2 proves that the policy performance difference between source and target domains is upper-bounded by the KL divergence summation over latent states in the generative diffusion trajectories.
• DADiff-modify achieves an average 8.7% improvement over competitive baselines across 16 MuJoCo tasks with various shifts, with up to 42.3% gain on halfcheetah (broken back thigh) task.
• DADiff-select matches or exceeds state-of-the-art VGDF performance on challenging shifts (e.g., halfcheetah no thighs, hopper big head) while being 3× faster to train.
• Both DADiff variants maintain stable performance across diverse domain shifts (kinematic, morphology, friction, gravity) unlike baselines VGDF, PAR, DARC which fluctuate strongly.
• DADiff-modify incurs only a modest GPU memory increase (~10–20 MB more than PAR/DARC) while achieving higher adaptation performance.
• Parameter studies show penalty coefficient (λ) and selection ratio (ξ%) critically influence performance and must be tuned per task for best results.
• DADiff outperforms PAR significantly in stochastic environments simulated by adding noise with std ς up to 0.03, demonstrating robustness where PAR's one-to-one latent assumption breaks down.
• Reward distribution analysis shows that DADiff-select preserves low-reward source data important for guiding policy, explaining its strength where DADiff-modify fails.

Threat model

The adversary corresponds to domain shift introducing dynamics mismatch between source and target environment transition functions. The adversary cannot change state/action spaces or reward functions, nor can it manipulate or manipulate agent internals. The method assumes only limited target domain interactions are possible; the adversary enforces a distributional shift that challenges direct policy transfer. The defense models and adapts under these constraints without assuming knowledge of target dynamics.

Methodology — deep read

1. Threat Model & Assumptions: The authors consider a standard online dynamics adaptation setup for cross-domain RL, where the state space, action space, and reward function are consistent across source and target domains, but the transition dynamics differ. The policy is trained with abundant source-domain data and limited target-domain interactions (e.g., 1M steps in source and 105 steps in target). The adversary is essentially the domain shift introducing dynamics mismatch. The approach assumes no prior knowledge or access to high-fidelity simulators of the target domain. 2. Data: Experiments use standard continuous control locomotion tasks from MuJoCo Gym environments (ant, hopper, halfcheetah, walker). The source domain is the original environment; target domains include systematic shifts in morphology, kinematics, friction, and gravity. Each method is run with five random seeds; 1 million source steps and limited (105) target steps are allowed for training. The dataset includes replay buffers from both domains with tuples (state, action, reward, next state). 3. Architecture/Algorithm: The core novel component is leveraging denoising diffusion probabilistic models (DDPM) conditioned on (state, action) pairs to model the distribution of next states. The diffusion model defines a generative trajectory as a sequence of latent noisy states leading to the next clean state. Two noise models are trained: ϵ_src (known noise from forward diffusion) in source domain and ϵ_θ_tar in target domain trained with limited samples to approximate target noise. The discrepancy between source and target latent state transition distributions is computed as KL divergence, which reduces to weighted squared noise prediction differences (Equation 8-10). The discrepancy score d(s,a,s') aggregates this KL over all diffusion timesteps K. Two variants exploit d(s,a,s'): (i) DADiff-modify penalizes source domain rewards by λd to adjust value function updates, and (ii) DADiff-select filters source data batches to keep only fractions ξ% with smallest d. Both variants train soft actor-critic (SAC) style policies with policy and twin Q-function losses, incorporating samples from source and target replay buffers. 4. Training Regime: SAC-based training uses replay buffers from source and target domains, with target data collected every F=10 steps. The target diffusion noise model is trained each iteration with samples from the limited target buffer. The penalty coefficient λ or selection ratio ξ%, diffusion step count K, and batch size N are hyperparameters studied. Training runs on GPUs with multiple seeds; typical training steps are 1 million environment steps in source with 105 target steps. 5. Evaluation Protocol: Performance is measured as cumulative returns after adaptation across 16 domain shift tasks. Baselines include DARC, VGDF, PAR, SAC variants, and Oracle SAC trained extensively in target domain. Metrics include mean return ± standard deviation over five seeds, GPU memory and runtime costs. Ablations study hyperparameters λ, ξ%, and K. Robustness is tested in stochastic dynamics simulated by action noise of various standard deviations ς. 6. Reproducibility: The authors release code at https://github.com/hanyang-chen/DADiff-release allowing implementation replication. The MuJoCo environments are publicly available benchmarks. Detailed hyperparameters and algorithms are provided, including pseudocode in Algorithm 1. Overall workflow: Collect source domain data and limited target data; train target noise diffusion model; estimate latent state discrepancy d(s,a,s') using the forward DDPM process; use d either to penalize source rewards or filter source data; update SAC actor and critic networks jointly on combined data. Over successive iterations, the policy adapts to minimize dynamics mismatch, improving target domain performance. The most computationally intensive step is training diffusion noise models per target batch, which is efficient due to parallel latent state generation. That completes the core methodology.

Technical innovations

• Framing dynamics mismatch in cross-domain RL as a discrepancy between generative diffusion model latent trajectories, enabling fine-grained divergence measurement.
• Deriving a theoretical performance bound linking policy performance gaps to KL divergences over latent state transitions in diffusion trajectories (Theorem 4.2).
• Introducing a practical discrepancy metric computable from differences in diffusion noise predictions across source and target domains without full trajectory sampling.
• Developing two complementary adaptation variants—reward modification and data selection—based on diffusion-driven discrepancy to improve policy transfer.
• Extending prior work (e.g., PAR) by removing the restrictive one-to-one latent state mapping assumption, enabling applicability to stochastic domains.

Datasets

• MuJoCo locomotion tasks (ant, hopper, halfcheetah, walker) — standard benchmarks used with simulated morphology, kinematics, friction, and gravity shifts — publicly available through OpenAI Gym

Baselines vs proposed

• DARC: average return across sixteen tasks lower than DADiff-modify by ~8.7%
• VGDF: competitive on some tasks (friction shifts) but 3× slower training runtime than DADiff-select
• PAR: significantly worse adaptation performance in stochastic domains compared to DADiff-modify
• SAC-IW, SAC-tune: inferior or unstable performance versus DADiff variants
• Oracle SAC (trained with 1M target steps): DADiff-modify achieves returns comparable to Oracle on many tasks

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16090.

Fig 1: Illustration of DADIFF. This figure visualizes the generative trajecto-

Fig 2: Adaptation performance under kinematic, morphology, friction, and gravity shifts (from top to bottom). The solid curves and the shaded regions

Limitations

• Performance sensitive to hyperparameter tuning of penalty coefficient λ and data selection ratio ξ%, requiring task-specific tuning.
• Diffusion noise model capacity limits performance gains beyond ~100 diffusion timesteps due to estimation errors.
• Experiments restricted to MuJoCo locomotion tasks; generalization to other RL domains (e.g., discrete action or vision-based states) is untested.
• Limited study under extreme or adversarial distribution shifts; robustness beyond simulated shifts is unclear.
• Computational overhead from latent state generation and target noise model training modest but non-trivial relative to simpler baselines.

Open questions / follow-ons

• How does the diffusion-based discrepancy estimation scale to high-dimensional or partially observable state spaces common in vision-based RL?
• Would end-to-end training jointly optimizing policy and diffusion noise model yield further adaptation improvements?
• Can the latent diffusion trajectory framework be extended to discrete action spaces and multi-agent RL settings effectively?
• What are the theoretical and practical impacts of applying this method to real-world robotics tasks with complex stochasticity and sensor noise?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners working in adversarial or cross-domain generalization environments, DADiff exemplifies a theoretically sound and empirically validated method to detect and adapt to subtle distributional shifts in environment dynamics. The fine-grained diffusion trajectory discrepancy metric could inspire new ways to quantify distribution mismatches in user behavior or bot simulation models beyond coarse classification or representation gaps. Additionally, the dual adaptation strategies (reward modification vs data selection) highlight trade-offs between penalizing suspicious behavior versus filtering training data to maintain robustness. Practitioners should consider such generative model-based discrepancy estimation approaches when traditional metrics fail to capture nuanced temporal divergences in user interaction dynamics, especially when target behavior data is limited. However, the complexity and computational cost of diffusion modeling require careful assessment of feasibility in production CAPTCHA or bot-detection pipelines.

Cite

@article{arxiv2607_16090,
  title={ DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning },
  author={ Hanyang Chen and Anirudh Satheesh and Longchao Da and Hua Wei },
  journal={arXiv preprint arXiv:2607.16090},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16090}
}

Read the full paper

• arXiv:2607.16090 (abstract)
• arXiv:2607.16090 (PDF)