Masked Visual Actions for Unified World Modeling

Source: arXiv:2607.19343 · Published 2026-07-21 · By Hadi Alzayer, Wenlong Huang, Haonan Chen, Christopher Luey, Lvmin Zhang, Maneesh Agrawala et al.

TL;DR

This paper addresses the challenge of integrating low-level robot actions into pretrained video models to enable unified forward and inverse robotic world modeling. The authors propose Masked Visual Actions (MVA), a novel pixel-space representation that expresses action as a partially revealed trajectory of entities (e.g., robot arms or objects) directly within the video input space. By conditioning the video model on revealed robot motion, it acts as a forward dynamics model predicting scene response; by conditioning on revealed object motion, it functions as an inverse model recovering corresponding robot behavior. Finetuning a large-scale pretrained video diffusion model on only 15 hours of combined real and simulated masked trajectory data, they achieve strong visual fidelity and controllability across a variety of robots and scenes, including unseen robot embodiments. The single finetuned checkpoint supports multiple downstream applications, such as policy evaluation, model-based planning via rollouts, and inverse modeling for robot action extraction from desired object motions.

Key findings

• Masked Visual Actions conditioning reduces LPIPS from 0.362 (Ctrl-World baseline) to 0.0945 on DROID dataset, with SSIM improvement from 0.708 to 0.887 and PSNR from 18.15 to 23.74.
• On BEHAVIOR dataset with an unseen bimanual robot, Masked Visual Actions outperform Ctrl-World in LPIPS (0.123 vs 0.196), SSIM (0.843 vs 0.837), and PSNR (22.90 vs 18.39), demonstrating strong generalization.
• Using masked visual actions improves model-based planning success rates across six diverse manipulation tasks by 7-26% over base policy execution.
• Rollouts produced by the video model correlate highly with real-world task success rates (Pearson r=0.982) in RoboCasa simulations.
• In a real-world setup with 4 manipulation tasks, video-model simulated progress closely matches executed task progress distributions, with slight positive bias.
• Masking robot trajectories as pixel-aligned inputs generalizes zero-shot from forward conditioning (robot masked) to inverse conditioning (object masked), allowing single model use for both forward and inverse planning.
• When comparing action representations, masked visual actions outperform sparse signals like end-effector pose or skeleton visualization in out-of-distribution generalization to new robots.
• Inverse modeling via masked visual actions plus learned inverse dynamics yields 90% success on COFFEESERVEMUG task, exceeding baselines trained specifically on this task.

Threat model

The system assumes an adversary without the ability to directly manipulate or corrupt the video input or masking procedures. Adversaries do not have access to the internal states of the pretrained video model aside from conditioning inputs. The method does not address adversarial attacks on the video model or masking representation, nor does it consider physical adversarial behaviors that could break model assumptions. The threat model is limited to benign forward/inverse modeling of robot actions in standard scenarios.

Methodology — deep read

The authors use a large pretrained video diffusion model (Wan-Fun-Control 2.2 14B) as the backbone. Their key idea is to represent robot and object trajectories as masks over the pixel space of input videos, thus conditioning the video model on partially revealed spatiotemporal pixel trajectories (masked visual actions). This visual conditioning aligns actions directly with the model's pretrained video domain, making the representation dense and embodiment-agnostic.

The threat model involves learning a forward dynamics function that predicts object movement given robot actions, and an inverse dynamics function that recovers robot motion given desired object outcomes. The model is trained only on forward examples (robot masked), but at test time, it supports both forward conditioning (predict passive entities from active robot trajectory) and inverse conditioning (predict active robot trajectory from passive object trajectory) without retraining.

Data comes from two main sources: (1) DROID, a real-world robot manipulation video dataset with segmentation masks from SegmentAnything; and (2) Robocasa, a simulated dataset. Video frames are paired with masked trajectories of robot arms and objects, either segmented or rendered via known URDF mesh with translucent rendering and highlighted grippers.

During training, a binary mask per frame indicates which pixels (corresponding to robot or object trajectories) are revealed versus to be predicted. The model inputs the masked video frames along with an initial reference frame. For missing pixels, a uniform gray background is used.

The video model's masked conditioning video is encoded via the model's standard autoencoder; conditioning signals are concatenated spatially to the latent code before diffusion. Finetuning is done via LoRA with rank 256, batch size 4, over approximately 10,000 steps on 8 NVIDIA H200 GPUs (~4 days). Hyperparameters beyond LoRA rank and batch are not detailed.

Evaluation uses common video synthesis metrics: LPIPS, SSIM, PSNR on held-out scenes from DROID (seen robots), BEHAVIOR (unseen bimanual robot), and custom real-world data with a modified end-effector. Additional downstream tasks are policy evaluation (correlating rollout success with ground-truth), model-based planning (selecting best action sequences via rollout scoring), and inverse modeling (synthesizing robot actions for desired object trajectories, then extracting low-level commands). Baselines include Ctrl-World (end-effector state conditioning), Wan-Move (trajectory conditioning), image-to-video, and imitation learning methods.

A concrete example: The model receives an initial image and a masked video showing the robot arm trajectory over time. It generates the full video including object responses. Alternatively, masking the object trajectory instead cues the model to generate robot motion achieving the manipulation. Evaluation shows LPIPS improvements and demonstration rollout success gains. The inverse dynamics model extracts robot commands from generated robot videos without explicit inverse training.

The authors promise code, data, and weights release for reproducibility, but these are not presently available from the paper itself.

Limitations include dependency on the pretrained video model's capabilities, correlation rather than causation learning, inference speed limits, and that the simulated training data may not cover all real-world distribution shifts.

Technical innovations

• Recasting robot and object actions as pixel-space masked trajectories to align control signals with pretrained video model representations.
• Unified forward and inverse world modeling via flexible conditioning on masked active or passive entity trajectories within the same video model checkpoint.
• Efficient adaptation of a large pretrained video diffusion model using LoRA finetuning on only ~15 hours of combined real and simulated masked data.
• Robust generalization to unseen robot embodiments and out-of-distribution end-effectors by dense, visual pixel-aligned action conditioning.

Datasets

• DROID — several hours of real-world robotic manipulation video with segmentation masks — public
• Robocasa — simulated robotic interaction dataset with paired video and robot states — public
• BEHAVIOR — bimanual robot manipulation dataset, used for unseen embodiment testing — public
• Custom real-world data with modified Franka Emika Panda end-effector — non-public

Baselines vs proposed

• Ctrl-World [25]: LPIPS=0.362 vs MVA: 0.0945 on DROID
• Ctrl-World [25]: PSNR=18.15 vs MVA: 23.74 on DROID
• Ctrl-World [25]: LPIPS=0.196 vs MVA: 0.123 on BEHAVIOR (unseen robot)
• Wan-Move [13]: LPIPS=0.534 vs MVA: 0.0945 on DROID
• Image-to-Video [63]: LPIPS=0.521 vs MVA: 0.0945 on DROID
• Planning improvement: +7-26% success over base policies across 6 manipulation tasks via MVA rollouts
• Inverse modeling success: MVA + inverse dynamics = 90% success vs Diffusion Policy 85%, ACT 80%, SmolVLA 50% on COFFEESERVEMUG task

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.19343.

Fig 1: Masked Visual Actions. We finetune a video model to condition on masked trajectories of robots,

Fig 2: Comparing action representations for learning. Low-dimensional robot actions are compact, but

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The model learns correlation between motions rather than causal physical interactions, limiting interpretability for control.
• Inference speed and complexity are constrained by the underlying large-scale video diffusion backbone, limiting real-time use.
• Training data covers only 15 hours of combined real and simulated videos, which may limit robustness to out-of-distribution scenarios beyond those tested.
• The rendering-based conditioning requires accurate robot state and camera calibration, limiting applicability to datasets without these annotations.
• The positive bias in imagined rollouts (overestimating task progress) suggests imperfect fidelity for policy evaluation.
• The inverse modeling pipeline relies on a separately trained inverse dynamics model to extract executable robot commands.

Open questions / follow-ons

• How to extend masked visual action conditioning to explicitly learn causal physical interactions rather than correlations?
• Can the approach be scaled to more complex multi-agent, multi-object manipulation scenes with richer interaction schemas?
• What are the limits of zero-shot inverse modeling generalization, and can iterative finetuning improve inverse model accuracy further?
• How would adversarial perturbations to masked conditioning or real-world noise affect forward and inverse world model robustness?

Why it matters for bot defense

For bot-defense practitioners, the core insight of pixel-aligned dense action conditioning could inspire novel input representations for challenges requiring joint modeling of actions and dynamic visual context. The idea to unify forward and inverse modeling within the same masking-conditioned video framework provides an efficient paradigm for predicting both action outcomes and inferring hidden agent behaviors from observed effects. Although this work focuses on robotic manipulation rather than user behavior or bot detection, its approach to unified visual action representation may inspire CAPTCHAs that better simulate and interpret interactive scenarios or physical manipulations. Moreover, the ability to generate realistic rollouts conditioned on partial visual inputs could be adapted for behavioral simulation in user interaction verification tasks. However, the computational cost and model complexity here may limit direct real-time CAPTCHA integration without substantial optimization.

Cite

@article{arxiv2607_19343,
  title={ Masked Visual Actions for Unified World Modeling },
  author={ Hadi Alzayer and Wenlong Huang and Haonan Chen and Christopher Luey and Lvmin Zhang and Maneesh Agrawala and Gordon Wetzstein and Li Fei-Fei and Yilun Du and Jiajun Wu and Jia-Bin Huang },
  journal={arXiv preprint arXiv:2607.19343},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.19343}
}

Read the full paper

• arXiv:2607.19343 (abstract)
• arXiv:2607.19343 (PDF)