Data Pyramid for Embodied Manipulation

Source: arXiv:2607.24744 · Published 2026-07-27 · By Yifan Ye, Yankai Fu, Yaoxu Lv, Bohan Hou, Jun Cen, Lingdong Kong et al.

TL;DR

This work addresses the critical question of data sourcing for embodied foundation models, which require richly annotated, physically grounded data linking visual observations, physical states, and robot actions. Unlike traditional multimodal models trained on large-scale internet vision-language data, embodied agents demand datasets that faithfully represent physical interactions and action consequences. The authors propose a systematic organization of the embodied data ecosystem as a "data pyramid" with five tiers, ordered by a trade-off between scalability and robot alignment: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. Each source is characterized along six axes: scalability, robot alignment, quality, diversity, reusability, and physical fidelity, highlighting their unique strengths and limitations. The work comprehensively surveys datasets in each category, reviews collection pipelines, sensing modalities, and embodiment choices, then analyzes recent embodied foundation models through the lens of their data recipes and the alignment challenges posed by heterogeneous sources. Finally, it identifies open challenges for data collection and integration, such as tactile data acquisition, failure and recovery data, cross-embodiment action alignment, and principled data recipe design.

The main contribution is this data-pyramid taxonomy that clarifies the complementary roles of diverse data sources for embodied AI, emphasizing the fundamental tension between physical robot fidelity (highest in real-robot data) and scalability (highest in general or simulated data). The analysis shows that state-of-the-art models increasingly incorporate heterogeneous data mixtures rather than relying solely on real-robot trajectories, enabling advances in perception, reasoning, planning, and action. The work also surfaces key future directions to scale up embodied datasets with richer supervision and broader embodiment transfer. Overall, the paper lays a foundation for more principled, data-centric design of next-generation embodied foundation models.

Key findings

• Embodied data categorized into five pyramid layers: real-robot, UMI-style, egocentric/exocentric, simulation, and general data, organized by scalability vs robot alignment (Fig 1).
• Six dimensions for characterizing data sources: scalability, robot alignment, quality, diversity, reusability, and physical fidelity.
• Real-robot data offers greatest robot alignment and physical fidelity but lowest scalability, requiring hardware, operators, and environment resets.
• UMI-style uses handheld grippers without robot in loop, offering intermediate alignment and improved scalability.
• Egocentric/exocentric human videos capture rich, real-world interactions with high diversity and physical fidelity but no direct robot supervision.
• Simulation data provides scalable, parallel data generation with privileged information and labels but suffers from sim-to-real gaps in physics.
• General vision-language data offers vast scale and semantic richness but no grounding in physical action or contact dynamics.
• State-of-the-art embodied models like RT-2, GR00T, Motus increasingly mix data from multiple pyramid layers for improved capabilities in perception, planning, and action generation (Fig 3).
• Open challenges include collecting large-scale tactile datasets, failure/recovery data, scalable pipelines, cross-embodiment action alignment, exploitation of egocentric data for dexterous manipulation, and principled data recipes.

Methodology — deep read

The paper primarily provides a systematic survey and taxonomy rather than proposing a single new algorithm. The methodology involves several key steps:

1. Threat model & assumptions: N/A for security threat but assumes the need for diverse, scalable embodied data to train generalist robot models across different embodiments and tasks.

2. Data: The authors curate a comprehensive collection of datasets spanning five categories: real-robot datasets (e.g., RoboTurk, RoboNet), UMI-style datasets (e.g., Fast-UMI, DexUMI), egocentric/exocentric human videos (e.g., Ego4D, Charades-Ego), simulation datasets (e.g., ManiSkill, RoboVerse), and large-scale general vision-language corpora (e.g., RefCOCO, RoboVQA). Dataset sizes range from a few thousand demonstrations to millions of hours or millions of QA pairs. Labels vary from robot states and actions to human hand poses, multimodal signals, and semantic annotations.

3. Architecture / algorithm: Rather than introducing a novel model, the paper analyzes recent embodied foundation model families including embodied brain models, vision-language-action (VLA) models, and world-action models. It examines how these models select and combine data sources during pretraining, align action spaces across embodiments, and transform heterogeneous observations into unified representations.

4. Training regime: The surveyed models use varied training regimes incorporating multiple heterogeneous datasets. Training considerations include action-space alignment strategies, geometric alignment of observations from different camera viewpoints, and balancing robot-specific versus scalable data.

5. Evaluation protocol: The paper reviews downstream capabilities of models linked to data composition, including perception, spatial-temporal reasoning, affordance understanding, planning, and action generation. It summarizes how dataset properties correlate with model strengths in these areas.

6. Reproducibility: The paper provides a maintained open-source GitHub repository curating datasets and relevant resources to encourage community adoption and reproducibility. Specific model codes or weights are outside the scope.

A concrete example discussed is how real-robot demonstrations (high robot alignment but low scale) can be supplemented by simulation data (high scale but domain gap) and egocentric videos (human interaction variety) to train a VLA model like RT-2 that is capable of perception, instruction grounding, and manipulation across robot platforms. Action-space alignment modules bridge the different control parameterizations, and geometric alignment modules unify viewpoint discrepancies.

Technical innovations

• Formulation of a data-pyramid taxonomy organizing embodied datasets by scalability and robot alignment, supplemented by axes for quality, diversity, reusability, and physical fidelity.
• Detailed characterization and comparison of five key embodied data sources capturing complementary trade-offs in collection cost, embodiment closeness, and data semantics.
• Comprehensive analysis linking embodied foundation-model architectures to their data "recipes" and the alignment challenges of heterogeneous embodied data.
• Identification of six future challenges for embodied data including tactile dataset construction, failure/recovery data, scalable pipelines, cross-embodiment alignment, and principled data recipes.

Datasets

• Pinto and Gupta (2015) — 50K demonstrations — Real-robot Single-arm grasping
• RoboNet (2019) — 162K demonstrations / 833.3 hours — Real-robot Single and multi-arm manipulators
• Fast-UMI (2024.09) — 9K demonstrations — UMI handheld gripper data
• DexUMI (2025.05) — 1.8K demonstrations — UMI handheld gripper data
• Ego4D (2021.10) — 3.6K hours — Egocentric human video data
• Charades-Ego (2018.04) — 68.8 hours — Egocentric human video
• ManiSkill 3 (2024.10) — 33K episodes — Simulation physics-based manipulation
• RoboVerse (2025.04) — 510K demonstrations — Simulation environment
• RefCOCO (2016.08) — 142K image-text pairs — General vision-language
• RoboVQA (2023.11) — 829.5K QA pairs — General vision-language

Baselines vs proposed

• Early embodied foundation models (e.g., PaLM-E 2023.3) trained primarily on real-robot trajectories, enhancing action fidelity but limited scale.
• Recent models like GR00T N1.7 (2026.4) and InternVLA-A1 (2026.1) combine real-robot, simulation, egocentric, UMI, and general data sources, achieving broader perception, reasoning, and action generation capabilities.
• Vision-language-action models trained on mixed data outperform those trained only on real-robot data by improved zero-shot transfer and reasoning (quantitative details not specified).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24744.

Fig 1: Overview of Organization and Scope. This work is organized into three parts. We first introduce the embodied

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• No novel empirical model or evaluation results; study is largely analytic and survey-based.
• Physical fidelity and sim-to-real gaps remain substantial challenges, not fully solved.
• Action and observation alignment across heterogeneous embodiments remain imperfect and active research areas.
• Lack of large-scale tactile and failure case datasets constrains learning contact-rich and robust manipulation.
• Human video datasets require complex retargeting for robot control, limiting direct supervision utility.
• Scalability of real-robot data collection is naturally limited by hardware and human operator resources.

Open questions / follow-ons

• How to scale tactile data collection and incorporate tactile sensing into embodied foundation models effectively?
• What are robust methods for collecting and integrating failure and recovery data to improve robustness and generalization?
• How can action representations and state alignments be standardized to better transfer data and policies across diverse robot embodiments?
• What principled data mixtures or recipes optimally combine pyramid layers for different embodied model architectures and downstream tasks?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners working at the intersection with embodied AI, this paper offers a crucial data-centric framework for understanding and constructing datasets that ground physical interaction in multimodal learning systems. The embodied data pyramid clarifies the inherent trade-offs between data scalability and robot alignment, which directly inform the design of embodied models capable of robust physical interaction. Capturing real-world manipulation dynamics is analogous to securing systems sensitive to physical-world interactions or detecting bots based on rich multimodal behavioral signals. Practical implications include careful curation of diverse data sources, aligning heterogeneous sensor/action modalities, and prioritizing scalable yet physically grounded data collection pipelines. The taxonomy and analysis guide developers in choosing appropriate data for training embodied agents capable of nuanced perception, planning, and action generation — all relevant for defending against increasingly sophisticated embodied or sensor-enabled bots.

Cite

@article{arxiv2607_24744,
  title={ Data Pyramid for Embodied Manipulation },
  author={ Yifan Ye and Yankai Fu and Yaoxu Lv and Bohan Hou and Jun Cen and Lingdong Kong and Duo Zheng and Tianxing Chen and Jiaming Liu and Ziang Cao and Yunfan Lou and Wei Chow and Xian Sun and Yingshuo Wang and Kuangzhi Ge and Xiaowei Chi and Xidong Zhang and Zhibo Pang and Yiwu Zhong and Sirui Han and Zhihe Lu and Weihao Yuan and Qifeng Chen and Michael Yu Wang and Yao Mu and Ziwei Liu and Jianfei Yang and Ping Luo and Shanghang Zhang },
  journal={arXiv preprint arXiv:2607.24744},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24744}
}

Read the full paper

• arXiv:2607.24744 (abstract)
• arXiv:2607.24744 (PDF)