WorldSample: Closed-loop Real-robot RL with World Modelling

Source: arXiv:2607.02431 · Published 2026-07-02 · By Yuquan Xue, Le Xu, Zeyi Liu, Zhenyu Wu, Zhengyi Gu, Xinyang Song et al.

TL;DR

WorldSample addresses the core bottleneck of real-robot reinforcement learning (RL): the high cost and limited coverage of physical rollouts which yield only a single trajectory per execution. It introduces a closed-loop data augmentation framework that tightly couples real robot rollouts with an action-conditioned world model to generate high-fidelity synthetic trajectories. This enables the expansion of limited real experiences into diverse counterfactual transitions grounded in physics and real observations, thus amplifying training data coverage without incurring additional physical interaction costs. To safely incorporate this synthetic data, WorldSample proposes Policy-Paced Learning (PPL), a mechanism that selectively filters and schedules synthetic samples based on Q-value estimates and policy uncertainty to mitigate value overestimation and hallucination noise.

Extensive real-world experiments on contact-rich and precise manipulation tasks demonstrate that WorldSample improves policy success rates by 28% on average while reducing real robot training steps by 59% compared to strong baselines including human-in-the-loop RL and prior world-model methods. Additionally, the framework significantly enhances world model visual fidelity (19.4 dB PSNR and 0.47 SSIM improvement) through online rollout adaptation. Ablations confirm PPL as critical to stabilizing learning with synthetic data. Overall, WorldSample provides an effective strategy to expand real-robot RL training efficiency and policy capability by tightly integrating real interaction, learned world dynamics, and principled synthetic data usage.

Key findings

• WorldSample improves average success rate across 5 real manipulation tasks from 56% (HIL-SERL baseline) to 82%, a 28% absolute increase (Table 1).
• Training steps are reduced by 59%, from 56K to 23K steps on average, and wall-clock training time reduces from 83 to 64 minutes, a 23% decrease.
• On the difficult Insertion task, WorldSample achieves 95% success with 50% fewer real interaction steps, and on Sorting 95% success with 33% fewer steps.
• World model visual fidelity improves substantially through online rollout adaptation, increasing PSNR from 8.5 to 28.4 and SSIM from 0.33 to 0.91 compared to pretrained demo-only (Table 2.a).
• Ablation shows removing Policy-Paced Learning (PPL) reduces success rate on insertion from 95% to 86%, with nearly 3x training steps needed; removing uncertainty scheduling further degrades success to 61%.
• Q-aware sample selection in PPL prevents overly optimistic or pessimistic value biases by balancing positive and negative synthetic trajectories within a Q-value margin δQ (Eq. 5).
• Uncertainty-guided data scheduling modulates the influence of synthetic data by policy entropy, reducing synthetic data when policy uncertainty is high to minimize value overestimation (Eqs. 6–9).
• WorldSample’s real-synthetic closed loop asynchronously updates the world model from real rollouts while generating synthetic trajectories in parallel to avoid blocking real-time robot control.

Threat model

The framework implicitly assumes a benign training setting where the primary challenge is the sample inefficiency and imperfect synthetic data quality from learned world models rather than adversarial agents. The main threat is value overestimation and hallucination noise caused by synthetic trajectories that do not perfectly reflect real dynamics. The approach assumes no malicious adversaries can manipulate real or synthetic data beyond the stochastic modeling errors inherent in the world model.

Methodology — deep read

1. Threat Model & Assumptions: The work assumes a physical robot agent operating in a manipulation environment with partial observability through multi-view RGBD cameras. The adversarial aspect is not emphasized; rather, the threat is the sample inefficiency and noise introduced by hallucinated synthetic data from imperfect world models. The adversary could be seen as stochastic model errors causing value estimation bias. The framework assumes access to initial human demonstrations but aims to expand beyond them.

2. Data: The data originates from a Galaxea A1X robot arm equipped with dual Intel RealSense D435i cameras (third-person and wrist views). Initial training uses 20 human demonstrations per task. Online data collection proceeds through physical rollouts with human interventions. Synthetic data is generated from a post-trained video prediction world model conditioned on real rollout trajectories plus locally perturbed counterfactual action sequences. The dataset includes both real trajectories (Dreal) and synthetic trajectories (Dsyn) with reward labels from an independent reward model.

3. Architecture/Algorithm: The core is a closed-loop real-synthetic data augmentation framework. A pretrained action-conditioned video prediction model (Cosmos-Predict2.5) is finetuned online with real rollout data to adapt to task-specific dynamics and visuals. Synthetic data is generated by sampling counterfactual perturbations around real action trajectories and rolling out future observations and rewards from the world model. Policy-Paced Learning (PPL) controls the incorporation of synthetic data using Q-aware sample selection (filtering synthetic transitions to balance positive and negative value signals within a Q-value margin) and uncertainty-guided scheduling (modulating synthetic data influence by policy entropy). The RL agent trains asynchronously from both real and synthetic buffers with separate weighting on losses.

4. Training Regime: Policies are initialized from demonstrations and trained online with mixed real and synthetic data in batches of size 256. The world model adapts asynchronously post-training from accumulated real rollouts. Experiments are conducted on five manipulation tasks each evaluated until convergence or matched wall-clock time. Ablations remove components of PPL to test impact. Training seeds and hardware details are not explicitly described in the paper.

5. Evaluation Protocol: Metrics include success rate, intervention rate, training steps, and wall-clock training time on real robot tasks. Baselines include HIL-SERL (human-in-the-loop RL), and two prior world-model co-training methods (VLAW and WMPO). The world model is evaluated on PSNR, SSIM, and LPIPS for visual fidelity on held-out real rollout data. Ablations test the role of Q-aware selection and scheduling. Training curves plot success and intervention rates against training steps. Statistical tests are not reported.

6. Reproducibility: The authors cite use of the publicly available Cosmos-Predict2.5 model and detail the robotic setup and camera views. Code and project page are referenced but no explicit release of code or trained weights is mentioned in the source. Full implementation details and hyperparameters appear in an appendix which is not included in the excerpt. The dataset is primarily proprietary real robot data.

Example end-to-end usage: Starting with 20 human demonstrations, the robot executes online physical rollouts under a policy. These rollouts train and adapt the world model asynchronously. The world model generates multiple synthetic, counterfactually perturbed trajectories conditioned on real rollouts’ initial observations and perturbed action sequences. These synthetic trajectories are filtered by Q-value and scheduled based on policy uncertainty to form mini-batches mixed with real data. The policy is updated from this augmented buffer, improving sample efficiency and allowing the robot to learn complex manipulation tasks faster with fewer real interactions.

Technical innovations

• Real-synthetic closed loop combining online real rollouts with asynchronously updated action-conditioned video world models to generate physically grounded synthetic data.
• Counterfactual trajectory generation via local perturbations of real action sequences to ensure physically feasible synthetic transitions.
• Policy-Paced Learning (PPL), combining Q-aware sample selection to balance synthetic positive and negative samples preventing value overestimation, and uncertainty-guided synthetic data scheduling modulated by actor policy entropy.
• Asynchronous pipeline design that allows continuous real robot control while world model generation and adaptation proceed in parallel.

Datasets

• Real-world manipulation dataset — ~20 human demonstrations per task plus online rollouts — proprietary robot data from Galaxea A1X platform

Baselines vs proposed

• HIL-SERL: average success rate = 56% vs WorldSample: 82%
• HIL-SERL: training steps = 56K vs WorldSample: 23K
• VLAW: average success rate = 64% vs WorldSample: 82%
• WMPO: average success rate = 69% vs WorldSample: 82%
• Pretrained world model PSNR = 8.5 vs rollout-adapted = 28.4
• Demo-only world model PSNR = 10.3 vs rollout-adapted = 28.4
• PPL ablation without scheduling success = 61% vs full PPL 95%
• PPL ablation without Q-selection success = 76% vs full PPL 95%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02431.

Fig 1: WorldSample overview. WorldSample expands real-robot rollout into multiple physically grounded

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 2: WorldSample pipeline. WorldSample closes a real-synthetic data loop between physical rollouts,

Fig 3: shows the corresponding online training curves. WorldSample generally reaches higher success

Fig 8 (page 4).

Limitations

• Focuses on single-task learning with a relatively fixed scene distribution; no evaluation on multi-task or generalization to diverse environments.
• Does not fully explore or evaluate robustness under distribution shift, adversarial conditions, or sensor noise.
• Synthetic trajectory quality depends heavily on the adapted world model’s accuracy; residual errors remain and could affect policy robustness.
• No reported statistical significance tests or confidence intervals for results.
• Reproducibility limited by lack of publicly available real robot data and undisclosed training seeds or hardware details.
• Human intervention remains part of the training loop; a fully autonomous pipeline is not demonstrated.

Open questions / follow-ons

• How well does WorldSample generalize to multi-task or multi-scene real robot learning scenarios?
• Can Policy-Paced Learning be extended or adapted for varying quality of world models across different tasks or domains?
• What is the impact of long-horizon synthetic trajectories or deeper imagined rollouts on performance and stability?
• How does WorldSample perform under adversarial perturbations or unexpected environment changes during real-robot deployment?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focused on real-world robotic interfaces or embodied AI, WorldSample offers an instructive example of how physical interaction data can be efficiently augmented with learned generative models to reduce expensive live interactions. Its focus on controlling synthetic data quality via principled filtering and scheduling highlights the importance of managing hallucination noise in any closed-loop system reliant on learned simulators or predictive models. While the domain is robotic manipulation rather than visual CAPTCHA recognition, the core insights into model-based data augmentation, uncertainty-guided training pacing, and real-synthetic co-training could inspire similar approaches to mitigate expensive human or bot querying costs in CAPTCHA learning or adversarial robustness settings.

Cite

@article{arxiv2607_02431,
  title={ WorldSample: Closed-loop Real-robot RL with World Modelling },
  author={ Yuquan Xue and Le Xu and Zeyi Liu and Zhenyu Wu and Zhengyi Gu and Xinyang Song and Bofang Jia and Ziwei Wang },
  journal={arXiv preprint arXiv:2607.02431},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02431}
}

Read the full paper

• arXiv:2607.02431 (abstract)
• arXiv:2607.02431 (PDF)