DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Source: arXiv:2607.08751 · Published 2026-07-09 · By Yunchao Yao, Zhuxiu Xu, Tianqi Zhang, Zixian Liu, Sikai Li, Zhenyu Wei et al.

TL;DR

DexVerse addresses a critical gap in robotic dexterous manipulation research by introducing a large-scale, modular benchmark that spans 100 diverse tasks across multiple robot arms and dexterous hand embodiments. Unlike previous benchmarks focused on gripper-based or narrow task sets, DexVerse evaluates general-purpose manipulation policies under challenging conditions that include multi-goal, long-horizon, contact-rich, and articulated-object tasks. It also incorporates configurable visual variations and multi-modal observations, enabling robust visuomotor generalization analysis. Alongside the environment, the authors provide a novel VR-based teleoperation interface and a dataset of 3,180 expert demonstrations collected across six different hands, supporting both imitation learning and reinforcement learning research. Benchmarking four representative state-of-the-art imitation and vision-language-action methods reveals significant challenges remain, with best mean success rates around 34%, and zero success on many contact-precision and tool-use tasks. This positions DexVerse as a comprehensive testbed for advancing general dexterous manipulation capabilities.

Key findings

• DexVerse includes 100 dexterous tasks grouped into 8 categories such as primitive, articulation, bimanual coordination, and long-horizon tasks, supporting 3 robot arms and 6 dexterous hands.
• A dataset of 3,180 VR teleoperation expert demonstration trajectories was collected with synchronized proprioceptive, RGB, depth, point-cloud, and state observations.
• Four baseline imitation learning methods (Diffusion Policy, 3D Diffusion Policy, OpenVLA, π0.5) trained on 950 episodes achieved only 19%-34% mean success rates across 19 benchmark tasks.
• 3D Diffusion Policy and π0.5 tied for highest mean success rate at 34%, outperforming Diffusion Policy (32%) and OpenVLA (19%) on DexVerse tasks.
• Internet-scale vision-language-action pretraining does not provide clear advantage for dexterous manipulation: π0.5 matches but does not surpass best from-scratch methods.
• Observation modality performance depends on task: 2D image+state better for pick-and-lift, point-cloud inputs improve tool use, language conditioning helps with articulated object and precision contact tasks.
• All methods fail on tight-tolerance precision tasks requiring fine contact (e.g., PushT, InsertPen, SlideUtilityKnife scored near zero success rates).
• No single baseline dominates across all task families, indicating the challenge of developing universal dexterous policies.

Methodology — deep read

The authors designed DexVerse, a modular benchmark of 100 dexterous manipulation tasks organized into 8 categories capturing diverse interaction patterns, complexity levels, and temporal horizons. Each task is formalized as a Markov Decision Problem with defined object sets (Ω), initial state distributions (S_0), observation (O) and action (A) interfaces, and objective success predicates (G).

DexVerse supports multiple robot embodiments, including 3 arms (Franka Research 3, UR10e, xArm 7) and 6 hands (e.g., Shadow Hand, Allegro Hand), representing varied kinematics, joint limits, and morphologies, instantiated modularly to decouple embodiment from task logic. Observation modalities include RGB, depth, point clouds, proprioception, and simulator states, with configurable visual randomizations of textures, lighting, and camera poses to probe visuomotor robustness.

Demonstration data was collected via an XR-based teleoperation system using Apple Vision Pro, leveraging human hand and wrist tracking with inverse kinematics and dex-retargeting for different hands. This yielded 3,180 demonstration trajectories: 55 per single-goal task (mostly Shadow Hand plus one demo per other hand) and 20 per long-horizon task. Demonstrations recorded synchronized action-state pairs and provide tools for exact replay to mitigate simulator drift.

For benchmarking, four representative policies were evaluated: Diffusion Policy (DP), 3D Diffusion Policy (DP3), OpenVLA, and π0.5 VLA transformers. Policies were trained with behavior cloning on a common set of 950 demonstrations (19 tasks × 50 episodes/task) and tested in closed loop for 50 rollouts/task. Success rates were the primary metric, with per-task and per-category analysis conducted. No explicit adversarial or distribution shift testing was reported.

The authors highlighted varied modality benefits across tasks (e.g., 3D inputs aiding tool use), limitations of pretrained VLAs in this domain, and the difficulty in precision contact and bimanual tasks. The environment and dataset are released publicly with modular configuration files to facilitate reproducibility and extension.

Technical innovations

• A unified modular benchmark combining 100 diverse dexterous manipulation tasks with multiple robotic arms and fully dexterous hands.
• Embodiment-agnostic environment design supporting interchangeable arm-hand combinations and configuration-driven task instantiations.
• Configurable visual variation mechanisms (textures, lighting, camera angles) integrated into the benchmark to assess visuomotor generalization.
• VR-based teleoperation pipeline enabling efficient multi-embodiment expert demonstration collection with synchronized multi-modal sensory data.

Datasets

• DexVerse Teleoperation Dataset — 3,180 demonstration trajectories — collected via VR teleoperation interface across 100 tasks and 6 hand embodiments.

Baselines vs proposed

• Diffusion Policy: mean success = 0.32 vs 3D Diffusion Policy (DP3) mean success = 0.34
• OpenVLA: mean success = 0.19 vs π0.5 mean success = 0.34
• π0.5 matches DP3 on overall success but excels in articulated and precision contact tasks where OpenVLA lags
• All baselines achieve zero or near-zero success on precision tasks like PushT and InsertPen

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08751.

Fig 1: Overview of DexVerse, a modular benchmark for multi-task, multi-embodiment dexterous

Fig 2: Visualization of selected tasks from the DexVerse environments.

Fig 3: Visualization of task progression of the 5 long-horizon tasks in DexVerse environments.

Fig 4: Modular Environment Architecture

Fig 5: Visual demonstration of embodiments and visual variation.

Fig 6 (page 4).

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• Benchmark evaluations focus on imitation learning methods; no extensive reinforcement learning or adversarial robustness testing reported.
• Collected demonstrations emphasize Shadow Hand, with limited samples for other hand embodiments, restricting some generalization claims.
• Visual variation was primarily synthetic and limited to textures, lighting, and viewpoints; real-world visual domain shifts remain untested.
• Sim-to-real transfer and real-robot validations are absent, limiting direct deployment insights.
• Precision contact and force-based control challenges remain unresolved, highlighting missing force sensory modalities and closed-loop contact feedback.
• No ablation studies on representation design or policy architecture; interpretation of baseline method differences limited.

Open questions / follow-ons

• How can policies explicitly model and leverage force/torque feedback or tactile sensing to improve precision contact tasks?
• What methods best facilitate transfer learning across diverse hand-arm embodiments within or beyond DexVerse?
• How do policies trained and evaluated on DexVerse transfer to real-world robotics tasks involving diverse visual and physical conditions?
• Can reinforcement learning or hybrid imitation-RL approaches close the success rate gaps observed in complex long-horizon and multi-goal tasks?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective focused on human vs. robotic manipulation differentiation, DexVerse offers a highly detailed and diverse manipulation benchmark illustrating the substantial challenges robots face in achieving human-level dexterous control under variable embodiment and visual conditions. The benchmark's provision of multi-modal sensory input variations, long-horizon, and complex interaction tasks underscores the difficulty for automated agents to replicate human dexterity robustly. Practitioners designing bot-detection systems could draw inspiration from DexVerse's approach to multi-task evaluation under embodiment and sensory shifts to develop nuanced behavioral signatures that reveal robotic control limitations. Additionally, DexVerse’s findings that current state-of-the-art policies still struggle with fine manipulations suggest opportunities to incorporate precision-contact and multi-stage interaction challenges in CAPTCHA systems where bots exhibit detectable failure patterns.

Cite

@article{arxiv2607_08751,
  title={ DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation },
  author={ Yunchao Yao and Zhuxiu Xu and Tianqi Zhang and Zixian Liu and Sikai Li and Zhenyu Wei and Feng Chen and Dihong Huang and Kechang Wan and Chenyang Ma and Shuqi Zhao and Shenghua Gao and Masayoshi Tomizuka and Yi Ma and Mingyu Ding },
  journal={arXiv preprint arXiv:2607.08751},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08751}
}

Read the full paper

• arXiv:2607.08751 (abstract)
• arXiv:2607.08751 (PDF)