GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions

Source: arXiv:2608.06332 · Published 2026-08-06 · By Chenghao Gu, Hanyang Yu, Jingbo Zhang, Haitao Lin, Wenyao Zhang, Jinghe Wang et al.

TL;DR

GeniWorld addresses a critical challenge in robotic manipulation world models: robust generalization and spatially precise action conditioning in diverse and unseen environments. Prior action-conditioned world models typically rely on low-dimensional numerical action vectors that entangle robot embodiment and scene dynamics, limiting interaction fidelity and zero-shot out-of-distribution (OOD) generalization. Leveraging pretrained video generative models, GeniWorld innovates by converting numerical robot actions into dense, spatially grounded visual action representations via URDF-based rendering. This decouples robot kinematics from environmental dynamics, improving modeling of robot-environment interactions and mitigating overfitting to fixed scenes.

GeniWorld employs a causal autoregressive video prediction framework with spatially aligned latent embeddings for observations and visual actions, enabling closed-loop robot interaction with both policies and human teleoperators. Despite training solely on limited fixed-scene data, GeniWorld achieves superior in-domain visual prediction fidelity and robust zero-shot generalization to highly randomized, unseen environments. Experiments demonstrate that GeniWorld-derived policy evaluations strongly correlate with real-world success and that leveraging GeniWorld to synthesize diverse manipulation trajectories significantly improves downstream policy performance and out-of-distribution robustness across varied spatial layouts, object instances, distractors, and lighting conditions. This work highlights the promise of spatially grounded interactive world models as scalable imagination spaces for robust robotic manipulation learning and evaluation.

Key findings

• GeniWorld outperforms baselines (Ctrl-World, IRASim, EnerVerse-AC) on RoboTwin2.0 benchmark for clean-to-clean and clean-to-random tasks, achieving LPIPS scores of 0.055 (vs 0.165 baseline) and 0.144 (vs 0.285 baseline) respectively (Table I).
• Under zero-shot OOD conditions with randomized scenes, GeniWorld maintains low FID of 13.08 and FVD of 20.15, whereas prior methods degrade substantially (e.g., IRASim FID 174.52, FVD 191.26).
• Visual action conditioning enables faster training convergence and higher generative quality early in training compared with numerical and skeleton-based action inputs (Fig. 4).
• With only 5 flow-matching sampling steps, GeniWorld achieves a ~10× inference speedup with only ~2% degradation in FVD, outperforming numerical-action baselines that degrade ~22% at low steps (Fig. 5).
• Policy success rates evaluated in GeniWorld positively correlate with real robot task success in both in-domain and OOD settings with visual distractions (Fig. 7a), proving robustness as a policy evaluator.
• Augmenting limited real demonstration data with GeniWorld-synthesized diverse trajectories improves manipulation success from 40.8% to 69.0% overall and yields significant gains across spatial rearrangement, distractors, novel objects, and lighting shifts (Fig. 9,11).
• Spatially grounded visual actions enable physically plausible robot-environment interactions, avoiding artifacts like contactless grasps common in skeleton-conditioned models (Fig. 3).

Threat model

The adversary is primarily the natural domain and distribution shift in robotic manipulation environments—unseen object appearances, layout changes, lighting variations, and background distractors—that cause brittle policy behavior and poor world-model generalization. The threat model assumes no malicious actor but challenges the model's ability to robustly predict visually accurate outcomes under spatiotemporal perturbations that were unseen during training.

Methodology — deep read

1. Threat Model & Assumptions: The primary adversary is the natural domain shift in robot manipulation environments—background changes, object variations, and layouts—that challenge generalist policies. The model assumes access to paired offline observation-action trajectories collected from a fixed scene but is tested on out-of-distribution scenarios without further adaptation. No adversarial or malicious agent is considered.

2. Data: The dataset is based on RoboTwin2.0 benchmark with 2250 training episodes and 250 held-out test episodes per task, covering 50 tasks with 121 frames per episode at 24 fps. Two evaluation splits: Clean-to-Clean (same distribution) and Clean-to-Random (OOD with varied objects, layouts, appearances). Four real-world tasks (Move Bowl, Fold Towel, Place Mug, Open Drawer) with teleoperated dual-arm Xtrainer robot collect limited demonstrations (25 per task) in clean tabletop settings, with some randomized object poses.

3. Architecture/Algorithm: GeniWorld converts numerical robot control actions into dense visual motion sequences by applying robot kinematics and URDF-based rendering from target camera viewpoints. The visual actions are encoded via a causal 3D variational autoencoder into latent tokens, spatially concatenated channel-wise with pretrained video latent tokens to form a joint representation. A causal autoregressive transformer with flow matching loss uses causal attention to predict next-frame video latents conditioned on this joint history, ensuring future frames depend only on past observations and encoded visual actions. Visual actions remain clean conditioning signals without noise added. KV caching accelerates autoregressive generation.

4. Training Regime: Training optimizes flow matching loss over batches sampled from offline dataset trajectories. The model is trained autoregressively to predict future observation sequences given past frames and current visual actions. During training, ground-truth frames are initially fed followed by model-generated frames to reduce exposure bias. Hyperparameters, batch sizes, epochs, and seed strategies are not explicitly detailed, but training convergence and comparison are evaluated over steps (iterations).

5. Evaluation Protocol: Evaluation uses PSNR, SSIM, LPIPS, FID, FVD, and EWMScore metrics for video prediction quality comparing generated and ground-truth sequences. Benchmarked against three prior methods in both clean and OOD settings. Policy evaluation correlates simulated policy success rates with real-world task outcomes using human annotation and vision-language model judges. Data synthesis benefits assessed by retraining policies augmented with GeniWorld-generated trajectories on multiple OOD conditions.

6. Reproducibility: The authors release code and provide a project page; datasets are either public RoboTwin2.0 or limited real-world demonstrations. While pretrained video diffusion backbone is used, the exact pretrained weights and training hyperparameters are not fully specified. The URDF and robot kinematic models are integrated with Isaac Sim for rendering consistency.

Example Walkthrough: Given an initial image from a fixed scene and a future numerical action sequence, GeniWorld first converts the numeric actions to rendered dense visual motion sequences via the robot's URDF and forward kinematics. These visual actions are encoded into latent tokens, concatenated with latent representations of the current observation frame, and fed into an autoregressive transformer which predicts the next video frame latent conditioned on history. By repeating this cycle, the model simulates future video sequences in closed-loop, enabling interaction with robot policies or human teleoperators. This process preserves spatial alignment between robot embodiment and scene dynamics, enabling accurate OOD generalization and realistic interaction synthesis.

Technical innovations

• Conversion of numerical robot actions into dense, spatially grounded visual action representations using URDF-based rendering to decouple embodiment kinematics from scene dynamics.
• Causal autoregressive video prediction model conditioned on spatially aligned visual action and observation latents, enabling explicit modeling of robot-environment interactions.
• Integration of a pretrained video generative diffusion backbone with minimal architectural modifications to leverage rich generative priors for interactive robot manipulation.
• Use of KV caching and flow matching loss for efficient, closed-loop inference enabling real-time autoregressive rollout conditioned on both robot policies and human teleoperators.
• Data synthesis pipeline combining instruction-driven image editing with interactive world-model rollouts to generate highly diverse manipulation trajectories from limited real demonstrations.

Datasets

• RoboTwin2.0 benchmark — 2,250 training and 250 test episodes per evaluated split — public
• Real-world dual-arm Xtrainer manipulation tasks dataset — ~25 demonstrations per task across four tasks — internal

Baselines vs proposed

• Ctrl-World: Clean-to-Clean LPIPS=0.165 vs GeniWorld (visual actions) LPIPS=0.055
• Ctrl-World: Clean-to-Random LPIPS=0.285 vs GeniWorld (visual actions) LPIPS=0.144
• IRASim: Clean-to-Random FID=174.52, FVD=191.26 vs GeniWorld FID=13.08, FVD=20.15
• Numerical-action conditioning: Clean-to-Clean PSNR=11.59 vs Visual Actions PSNR=27.57
• ControlNet-style conditioning: Clean-to-Clean PSNR=23.99 vs Visual Actions PSNR=27.57
• Policy success correlation: GeniWorld vs real-world shows strong positive correlation; Ctrl-World exhibits degraded robustness under distractors.
• Data-augmented policy: Real-only success rate 40.8% vs Real+Spatial+Diverse Gen 69.0%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.06332.

Fig 1: Overview of GeniWorld. GeniWorld is an autoregressive robotic world model that transforms action inputs into visual action representations,

Fig 2: Overview of the GeniWorld method. We convert robot actions into visual motions via URDF rendering. These motions are subsequently encoded

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The model is trained only on limited fixed-scene demonstrations, which may limit generalization to radically different robot morphologies or camera viewpoints.
• Evaluation focuses primarily on zero-shot OOD generalization to randomized scenes but does not test adversarial attacks or deliberate manipulator failure modes.
• Details on exact training hyperparameters and seeds are not fully specified, potentially impairing reproducibility of results by external researchers.
• Real-world demonstration datasets are small and task-specific, limiting ability to generalize results to broader manipulation domains without further data.
• Though leveraging pretrained video generative models accelerates training, reliance on such backbones may limit adaptation to highly novel visual domains without finetuning.
• The teleoperation pipeline performance is reported but not rigorously benchmarked under latency or network constraints typical in real-world remote operation.

Open questions / follow-ons

• How does GeniWorld perform when generalized to robots with different kinematic chains or sensor modalities beyond RGB video?
• Can the approach scale to tasks involving multi-agent or multi-robot collaborative manipulation within the same interaction space?
• What are the limits of zero-shot generalization when initial training data include multiple diverse scenes or entirely different lighting and camera viewpoints?
• How robust is GeniWorld to real-time teleoperation noise, latency, or incomplete action sequences during closed-loop interaction?

Why it matters for bot defense

For bot-defense and CAPTCHA-like systems involving physical robot environments or interactive robot interfaces, GeniWorld’s approach of leveraging spatially grounded, visual-action-conditioned world models offers a promising direction. Its ability to robustly model robot-environment interactions under distribution shifts may be translated to systems needing to detect or simulate human-robot manipulations in real or synthetic scenarios. Although primarily aimed at robotic policy learning, the implicit robustness to out-of-distribution changes could inspire improved CAPTCHA techniques that assess interactive reasoning under perturbed spatial or temporal conditions. Furthermore, GeniWorld’s closed-loop autoregressive design with efficient inference might serve as a model for real-time verification frameworks in simulated environments where maintaining interaction fidelity under limited samples and noisy inputs is critical. Bot-defense practitioners could adapt similar spatially grounded video prediction or generative simulation techniques to catch discrepancies between authentic human-driven manipulation and bot-generated action sequences.

Cite

@article{arxiv2608_06332,
  title={ GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions },
  author={ Chenghao Gu and Hanyang Yu and Jingbo Zhang and Haitao Lin and Wenyao Zhang and Jinghe Wang and Hanglei Jin and Shuzhao Xie and Jingyan Jiang and Zhi Wang },
  journal={arXiv preprint arXiv:2608.06332},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.06332}
}

Read the full paper

• arXiv:2608.06332 (abstract)
• arXiv:2608.06332 (PDF)