Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models

Source: arXiv:2607.05390 · Published 2026-07-06 · By Hongyu Li, Wanjia Fu, Xiaoyan Cong, Zekun Li, Binghao Huang, Hanxiao Jiang et al.

TL;DR

Deform360 addresses a critical gap in robotic manipulation research by providing a large-scale, real-world visuotactile dataset focused on deformable object dynamics. The problem tackled is that predicting the future states of deformable objects is highly challenging due to their complex material behaviors, high-dimensional state spaces, and occlusions during interaction. Existing datasets are limited in object diversity, sensory modalities, or fidelity. Deform360 overcomes these limitations by capturing 198 diverse daily-life deformable objects through 1,980 interaction sequences totaling over 215 hours, recorded simultaneously by 41 calibrated surround-view cameras and bimanual tactile grippers. This rich multi-view, visuotactile data is combined with a novel markerless 3D tracking pipeline that produces high-fidelity dynamic geometry and dense particle trajectories.

Using this dataset, the authors systematically benchmark state-of-the-art world models in two paradigms: 2D action-conditioned video models and 3D particle-based models. Their evaluation across frame, episode, and object generalization reveals a trade-off: physics-informed 3D particle models outperform video models in low-data regimes due to strong structural priors, whereas large pretrained video diffusion models generalize better zero-shot to new objects due to visual diversity learned at scale. They further demonstrate real-world applicability by using the learned 3D dynamics models for model predictive control (MPC) robot planning on deformable objects in distinct environments. In all, Deform360 sets a new benchmark and foundation for future work on generalizable deformable world models leveraging visuotactile inputs.

Key findings

• Deform360 dataset contains 198 objects, 1,980 interaction episodes, 41 camera views, 23.3M frames, and 215.7 hours of visuotactile data.
• 3D Gaussian Splatting achieves a reconstruction PSNR up to 30.00 dB and SSIM ~0.98 on 3D volumetric deformables.
• Visuotactile tracking reduces Chamfer distance error to 2.71×10^-5 m², 5x lower than visual-only tracking (1.41×10^-4 m²) under occlusions caused by grippers.
• Contact prediction model trained on visual streams achieves 88.67% accuracy and F1-score of 0.8909 versus random guessing (~50%).
• Physically-informed 3D particle model PhysTwin outperforms learning-based PGND and ParticleFormer in low-data per-episode reconstruction and future prediction (CD = 0.014 vs 0.032-0.039).
• At multi-episode generalization, 2D video model Cosmos achieves higher visual quality (PSNR=27.75) while 3D ParticleFormer yields lower Chamfer distance future prediction (CD=0.051 vs Cosmos not reported).
• In zero-shot multi-object generalization, Cosmos has better image metrics (PSNR=25.04) than 3D models but struggles with accurately following robot actions during long-horizon prediction.
• Robot planning using PhysTwin in MPC framework successfully manipulates deformable objects zero-shot in a different robot/environment setup.

Threat model

The 'adversary' is effectively the natural complexity of deformable objects exhibiting infinite degrees of freedom, severe occlusions, and contact-induced local deformations. The approach assumes the adversary cannot prevent multi-view camera coverage or tactile sensing access and does not consider deliberate attacks or spoofing. The main challenge is accurate perception and modeling despite these physical and sensory constraints.

Methodology — deep read

The paper's methodology follows a comprehensive pipeline from data collection to benchmarking:

1. Threat Model & Assumptions: The main adversary is the complexity of deformable objects with occluded, high-dimensional dynamics and contact interactions. The system assumes synchronized multi-view visuotactile input and bimanual tactile sensing. There's no explicit adversarial attack scenario; rather, the challenge is accurate modeling and tracking under occlusions and varied materials.

2. Data: Deform360 includes 198 real-world daily-life deformable objects categorized as 1D linear (ropes, cables), 2D thin-shell (fabrics, garments), and 3D volumetric deformables (plush toys, foam). It contains 1,980 interaction sequences (5 uni- and 5 bi-manual per object) captured by 41 HD cameras at 720x1280, 30 FPS, plus tactile sensing from bimanual UMI grippers synchronized at 30 Hz. The dataset totals 215.7 hours and 23.3 million frames, with thorough camera calibrations and robot proprioception.

3. Architecture / Algorithm: The authors use a markerless multi-view perception pipeline combining 3D Gaussian Splatting (3DGS) per-frame dynamic geometry reconstruction with 2D mask tracking (CoTracker3) lifted into 3D via calibrated depth maps. A physics-informed optimization fuses multi-view 3D tracks using tactile feedback enforcing temporal coherence, local rigidity (ARAP), spatial smoothness (Laplacian), and tactile velocity consistency. This yields high-fidelity tracked particle trajectories from occluded, deforming objects.

They benchmark 3D particle-based world models (PhysTwin—differentiable physics simulator with structural priors, ParticleFormer—a learned Transformer-based model, and PGND—a GNN-based learner) and 2D video diffusion world models (Cosmos-Predict 2.5B parameters) modified for action conditioning.

4. Training Regime: For 3D models, training uses particle trajectories with multi-step loss (Chamfer distance and tracking error) under different generalization regimes (frame, episode, object) with multiple episodes and objects held out for test. Video models are post-trained on Deform360 with the robot action inputs encoded as 6D poses and gripper openness. Hyperparameter details and exact seeds are not fully described; training hardware specifics are not indicated.

5. Evaluation Protocol: Three generalization regimes test reconstruction and future prediction:

• Per-episode: train on initial frames, predict unseen frames.
• Multi-episode: train on subset of episodes per object, test on unseen episodes.
• Multi-object: train on subset of objects, test zero-shot on unseen objects. Metrics include Chamfer distance and mean squared track error for 3D models, and image quality metrics (PSNR, SSIM, LPIPS) for rendered predictions. Contact prediction is evaluated with accuracy and F1-score. Robot planning success is qualitatively shown with MPC deployment.

6. Reproducibility: The dataset and project website (https://deform360.lhy.xyz) are publicly released. Model code and weights are not explicitly mentioned as released at time of paper. The dataset calibration and tracking pipeline details are described with some specifics deferred to appendix. Training and evaluation are reproducible given public data but some hyperparameters details remain unspecified.

Concrete example: For one interaction episode involving a fabric object, 41 synchronized videos are segmented and undistorted. 3D Gaussian Splatting reconstructs per-frame dynamic geometry. Using CoTracker3, 2D particle tracks are extracted in each view and lifted to 3D depth maps. These multi-view trajectories are fused using the optimization enforcing rigidity and tactile constraints, producing stable particle trajectories robust under self-occlusion by the robot grippers. These dense 3D particles serve as ground truth to train and evaluate the PhysTwin and ParticleFormer models. A 2D video diffusion model (Cosmos) is post-trained with corresponding RGB frames and robot actions. The trained models are then tested for prediction on future frames within the episode, on unseen episodes of the same object, and on unseen objects for zero-shot generalization.

Technical innovations

• A massive real-world visuotactile deformable object dataset with 198 objects, 41-camera 360° coverage, and synchronized bimanual tactile sensing far exceeding existing benchmarks in scale and modality.
• A novel markerless multi-view 3D tracking pipeline combining per-frame 3D Gaussian Splatting geometry with multi-view 2D tracking and physics-informed tactile fusion for robust particle trajectory extraction under occlusions.
• Systematic benchmarking of state-of-the-art 2D video diffusion models versus 3D particle dynamic models across multiple generalization regimes (frame, episode, object) using a comprehensive real-world deformable manipulation dataset.
• Preliminary deployment of learned 3D world models from the dataset in real-world model predictive control robot planning tasks, demonstrating generalizability across robot platforms and environments.

Datasets

• Deform360 — 23.3 million frames, 215.7 hours, 198 deformable objects, 1,980 interaction sequences — publicly released at https://deform360.lhy.xyz

Baselines vs proposed

• PhysTwin (physics-based 3D) reconstruction & future prediction CD: 0.014 / 0.014 vs PGND (learning-based 3D) 0.032 / 0.046 and ParticleFormer (learning-based 3D) 0.039 / 0.044 on per-episode tasks
• Cosmos (2D video model) PSNR reconstruction: 27.748 vs ParticleFormer 26.263 on multi-episode generalization
• ParticleFormer achieves lower future prediction Chamfer Distance 0.051 vs Cosmos (not reported) on multi-episode generalization
• Cosmos PSNR future prediction: 25.042 vs ParticleFormer 23.312 and PGND 22.049 on zero-shot multi-object generalization

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.05390.

Fig 1: Overview of Deform360. We collect a massive multi-view visuotactile

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Severe self-occlusion causes degradation in tracking quality when large object regions are unseen by most cameras for extended periods.
• Assumptions of local rigidity and smoothness in particle optimization may not hold for highly plastic or viscous materials, limiting tracking accuracy.
• Tactile sensors only measure normal-axis pressures and lack detection of tangential micro-slip, thus slip events are avoided manually but remain unmodeled.
• PhysTwin requires manual per-episode registration and thus is not evaluated for multi-episode or zero-shot generalization.
• Video models struggle to follow robot actions precisely in long-horizon predictions, limiting deployment for control without further fine-tuning or improved action conditioning.
• Details on training hyperparameters, random seed strategies, and hardware setups are sparse, potentially hindering exact experiment reproduction.

Open questions / follow-ons

• How to improve tracking and modeling for highly plastic or viscous deformable materials that violate local rigidity?
• Can tactile sensing modalities be extended to detect slip and friction to better inform tracking and dynamics modeling?
• How to scale 3D world models with strong structural priors to similar levels of zero-shot generalization as large-scale pretrained 2D video models?
• What are optimal action representations or reward functions to enable robust long-horizon planning with video-based world models?

Why it matters for bot defense

Although Deform360 is primarily focused on robotic manipulation and deformable object dynamics, several insights are pertinent to bot-defense and CAPTCHA learning analytics. First, the integration of multi-modal sensory data (visual + tactile) and dense 3D tracking can inspire richer user interaction modeling beyond purely 2D video analysis, improving robustness to occlusions or spoofing in CAPTCHA pipelines. Second, the benchmarking methodology contrasting explicit structural models (physics/graph-based) against large-scale implicit models (diffusion networks) parallels current trade-offs in bot-detection or user-behavior prediction tasks between interpretable models and large data-driven approaches. The dataset and evaluation protocols highlight the importance of comprehensive multi-view supervision and multi-regime generalization testing, which could guide the design of CAPTCHAs that are hard to model or predict for automated scripts but still generalizable for diverse human interactions. Finally, the challenges of occlusion, contact inference, and multi-scale generalization in this paper underscore the potential of leveraging auxiliary modalities or physics-informed regularizers to strengthen CAPTCHA resilience against evolving adversarial strategies.

Cite

@article{arxiv2607_05390,
  title={ Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models },
  author={ Hongyu Li and Wanjia Fu and Xiaoyan Cong and Zekun Li and Binghao Huang and Hanxiao Jiang and Xintong He and Yiqing Liang and Rao Fu and Tao Lu and Srinath Sridhar and Kevin A. Smith and George Konidaris and Yunzhu Li },
  journal={arXiv preprint arXiv:2607.05390},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.05390}
}

Read the full paper

• arXiv:2607.05390 (abstract)
• arXiv:2607.05390 (PDF)