dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models

Source: arXiv:2606.23623 · Published 2026-06-22 · By Yuhao Wu, Yitian Liu, Weijie Shen, Mishuo Han, Wenjie Xu, Haotian Liang et al.

TL;DR

This paper addresses the challenge of applying reinforcement learning (RL) to discrete diffusion Vision-Language-Action (dVLA) models for robotic manipulation. Current dVLA training relies heavily on supervised fine-tuning (SFT) but suffers from limitations due to distribution shift and lacks effective RL strategies for policy improvement. A fundamental difficulty is the intractability of computing the marginal probability of the final action in the multi-step denoising generation process, preventing direct application of standard policy-gradient methods like PPO. To overcome this, the authors propose dVLA-RL, a novel RL method that models the discrete diffusion denoising process as a Markov Decision Process (MDP) over denoising trajectories rather than final actions. This trajectory-level objective factors the joint probability of a sampled denoising path into stepwise transitions, thereby enabling a tractable PPO formulation and stable policy updates. Leveraging this formulation, the authors also develop a unified step scheduling strategy that dynamically adapts the number of denoising steps per task complexity, balancing success rates and computational efficiency. Extensive experiments on two robotic manipulation benchmarks, LIBERO and RoboTwin 2.0, show that dVLA-RL substantially improves over SFT-only baselines, achieving a 99.7% success rate on LIBERO and a 30.6% absolute improvement on RoboTwin 2.0, while remaining competitive with strong World-Action Model baselines. This work constitutes the first principled integration of RL with native discrete diffusion VLAs, unlocking their potential for real-world generalist robotics control.

Key findings

• dVLA-RL achieves a 99.7% average success rate on the LIBERO benchmark, surpassing its supervised fine-tuned (SFT) backbone MM-ACT* which achieves 88.1%.
• On RoboTwin 2.0 tasks, dVLA-RL improves the average success rate from 61.4% (SFT MM-ACT*) to 92.0%, a 30.6 percentage points gain.
• Largest per-task gains on RoboTwin 2.0 include +47.9 points on Handover Mic, +47.5 on Move Can Pot, and +43.1 on Lift Pot.
• Trajectory-level probability modeling significantly outperforms last-step marginal likelihood approximations, yielding higher success rates and more stable RL training (see Figure 3).
• Hybrid denoising-step scheduling tailored per task complexity improves computational efficiency while preserving or improving task success rates.
• dVLA-RL matches or exceeds state-of-the-art VLA and World-Action Model (WAM) baselines on LIBERO and RoboTwin 2.0, validating the competitiveness of native discrete diffusion policies with RL optimization.
• Gradient calculation is restricted to token predictions only, ignoring unmasking probabilities to maintain stable PPO optimization.
• dVLA-RL shows superior sample efficiency, achieving high success with fewer environment interactions compared to previous methods (Fig 1c).

Threat model

n/a — The paper focuses on reinforcement learning formulism and policy improvement for robotic manipulation and does not address security threat models or adversarial settings.

Methodology — deep read

The core methodological innovation is reformulating the discrete diffusion denoising action generation as a Markov Decision Process (MDP) over denoising trajectories (paths through intermediate token states). This avoids intractable marginalization over all possible denoising paths to compute the final action probability.

1. Threat Model & Assumptions: The agent operates within the standard robotic manipulation MDP with states comprising multimodal observations (vision, language, proprioception) and actions represented as fully denoised token sequences. The adversary scenario is not directly security-focused but considers environment stochasticity and challenges in precise control.

2. Data: The model leverages standard robotic benchmarks LIBERO and RoboTwin 2.0, using 500-1000 demonstrations per task for supervised fine-tuning initialization. Multitask joint training is performed rather than separate task-specific models.

3. Architecture/Algorithm: Starting from the MM-ACT discrete diffusion VLA backbone that generates action chunks through iterative masked denoising, dVLA-RL treats the sampled denoising path τ = (xK,...,x0) as the policy's trajectory. The joint likelihood pθ(τ|s) factorizes as the product of stepwise transitions, each transition decomposed into a token unmasking step determined by a Gumbel-TopK scheduler and categorical token generation probabilities.

4. Training Regime: PPO is adapted using a trajectory-level clipped surrogate objective that optimizes the probability of sampled denoising paths instead of the final marginal action likelihood. To stabilize RL updates, gradients on the discrete unmasking scheduler are dropped, focusing updates only on tokens that transition from [MASK] to actual tokens. Advantage estimates are computed at the chunk level, treating the entire denoising chain as a single environment-level decision nested with internal refinement steps. The RL training is conducted on the SFT-initialized policy.

5. Evaluation Protocol: Success rate is the primary metric, evaluated with tens to hundreds of trials across four LIBERO suites and eight RoboTwin tasks. Baselines include the SFT MM-ACT* backbone, other VLA methods, and WAMs. Ablations compare trajectory-level formulation to single-step marginal likelihood approximations, analyze different denoising step lengths, and evaluate a hybrid scheduling strategy assigning variable denoising horizons per task. Sample efficiency is measured by success rate vs environment interaction steps.

6. Reproducibility: The paper indicates the use of RLinf PPO implementation and publicly available benchmarks LIBERO and RoboTwin 2.0. The discrete diffusion MM-ACT backbone and code details are described, but explicit public code release or frozen weights are not stated in the text provided.

Example: For a given robotic state with visual and language inputs, the dVLA policy performs multi-step masked token denoising to generate an action sequence. During RL rollouts, the full denoising trajectory's joint log-probability is computed by multiplying stepwise transition probabilities. This trajectory-level likelihood is then used in the PPO objective to update policy parameters, improving task success probabilistically while retaining computational tractability.

Technical innovations

• Shift RL optimization target from intractable marginal action probability to tractable joint probability of discrete diffusion denoising trajectories.
• Formulate discrete diffusion denoising as a Markov Decision Process over trajectories, decomposing joint likelihood into product of stepwise transition probabilities.
• Introduce stable PPO policy gradients computed only over token generation probabilities, excluding discrete unmasking steps to avoid high variance.
• Develop a unified denoising-step scheduling strategy adapting trajectory length per task complexity for improved multi-task efficiency and performance.

Datasets

• LIBERO — size varies per subtask, standard single-arm robotic manipulation benchmark (Liu et al., 2023)
• RoboTwin 2.0 — multi-task, bimanual manipulation benchmark with domain randomization (Chen et al., 2025a)

Baselines vs proposed

• MM-ACT* (SFT baseline): average success rate 88.1% on LIBERO vs dVLA-RL 99.7%
• MM-ACT* (SFT baseline): average success rate 61.4% on RoboTwin 2.0 vs dVLA-RL 92.0%
• SimpleVLA-RL on LIBERO average 99.1% vs dVLA-RL 99.7%
• OpenVLA (Kim et al., 2025b) on LIBERO average 76.5% vs dVLA-RL 99.7%
• WAM-based LingBot-VA on RoboTwin average 97.4% vs dVLA-RL 92.0% (note different training settings)
• Trajectory-level RL objective improves success rate by up to ~10% over last-step marginal likelihood approximation on LIBERO (Fig 3)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.23623.

Fig 1: Overview of UDVLA-RL. (a) We present a unified probability modeling approach for

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 2: The overall pipeline of dVLA-RL. (1) Rollout & Action Generation: During inter-

Fig 8 (page 6).

Limitations

• Codebase and pretrained model release not explicitly stated, limiting exact reproducibility.
• Evaluation limited to simulation benchmarks LIBERO and RoboTwin 2.0; real-world robot experiments not reported.
• No adversarial or distribution shift robustness evaluation presented; performance under unseen environment variations unclear.
• The RL formulation assumes accurate advantage estimation on sparse binary rewards, which may limit scaling to more nuanced reward functions.
• Discrete unmasking step gradients are dropped, which may limit explicit scheduler improvement under RL.
• Comparisons to WAM baselines on RoboTwin note differing training settings, preventing strict controlled comparisons.

Open questions / follow-ons

• How does dVLA-RL perform and generalize in real-world robotic hardware with sensor noise and environmental variability?
• Can the trajectory-level RL framework be extended to incorporate richer reward functions or multi-objective optimization beyond sparse success signals?
• What are the theoretical convergence properties and limitations of ignoring gradients from the discrete unmasking scheduler under PPO?
• Could combining trajectory-level RL with offline reinforcement learning techniques improve sample efficiency or safety during physical deployment?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work is relevant in illustrating advanced techniques for RL over complex discrete generative processes with intractable marginal likelihoods. The formulation of trajectory-level likelihoods for multi-step discrete decoding presents a generalizable strategy for RL optimization where marginalization over latent generation paths is infeasible. Such insights could inspire novel CAPTCHA generation or verification mechanisms leveraging iterative discrete processes. Furthermore, the unified step scheduling approach to balance computational cost and refinement quality may inform adaptive security checks that modulate complexity according to client risk profiles. Although focused on robotic manipulation, the underlying principles of stable policy gradient optimization over discrete diffusion trajectory distributions have potential cross-domain applicability in sequential decision-making tasks relevant to bot detection and challenge design.

Cite

@article{arxiv2606_23623,
  title={ dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models },
  author={ Yuhao Wu and Yitian Liu and Weijie Shen and Mishuo Han and Wenjie Xu and Haotian Liang and Zhongshan Liu and Yinan Mao and Lei Xu and Xinping Guan and Ru Ying and Ran Zheng and Wei Sui and Xiaokang Yang and Wenbo Ding and Yao Mu },
  journal={arXiv preprint arXiv:2606.23623},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.23623}
}

Read the full paper

• arXiv:2606.23623 (abstract)
• arXiv:2606.23623 (PDF)