TableVerse: A Large-scale Tabletop Dataset with Real-world Grounded Layouts for Generalizable Manipulation

Source: arXiv:2607.21017 · Published 2026-07-23 · By Boyuan Wang, Yue Zhang, Xutao Xue, Xueyu Song, Yu Sun

TL;DR

TableVerse addresses a critical bottleneck in developing generalizable robotic manipulation policies by providing a large-scale, high-fidelity dataset of physically plausible tabletop scenes rebuilt from real-world images. Unlike prior synthetic or text-driven layout synthesis approaches that suffer from unrealistic object arrangements and collision issues, TableVerse introduces a deterministic Real2Sim pipeline that reconstructs 3D digital twins from unstructured internet media. This pipeline incorporates novel layout-preserving geometric optimizations and physics stabilization, enabling the generation of 100,000 unique, physically consistent tabletop environments with authentic object topologies, accurate metric scaling, and verified mechanical stability. Along with these environments, TableVerse synthesizes task-conditioned, collision-free pick-and-place trajectories, creating a richly annotated dataset to advance scalable policy learning for cluttered manipulation tasks.

The key innovation lies in converting noisy, occluded single-view internet snapshots into simulation-ready scenes through object extraction, composite asset decomposition, precise 6-DoF pose registration, and a three-phase Layout-Consistent Collision Rectification (LCCR) module that prevents mesh interpenetration before physics simulation. Experimental results on a 100-scene test set demonstrate TableVerse's superiority over SOTA baselines (MIDI, SAM3D, SceneMaker), with a 0% collision rate and significant gains in layout fidelity (7.14 vs ≤5.81), visual quality, and geometry quality scores measured by a multi-dimensional MLLM GPT-Score. Ablations validate the necessity of each pipeline stage, especially LCCR combined with physical settling under MuJoCo. The dataset spans nearly one million objects over 35,000 categories, with large-scale domain randomization to augment visual diversity. Despite some limitations in 3D reconstruction fidelity and inference speed, TableVerse establishes a new comprehensive benchmark for realistic, physically grounded tabletop scene synthesis and manipulation demonstrations.

Key findings

• TableVerse-100K dataset contains 100,000 unique tabletop environments with nearly 1 million object instances across 35,000 semantic categories.
• The TableVerse pipeline achieves 0.0% scene collision rate versus 72-90% for baseline single-view methods (Table 1).
• TableVerse achieves highest multi-dimensional GPT-Score: Layout Fidelity 7.14, Visual Quality 7.08, Geometry Quality 7.03, outperforming MIDI (5.81,5.08,4.83), SAM3D (5.73,5.44,5.47), and SceneMaker (4.90,5.48,4.85).
• The Layout-Consistent Collision Rectification (LCCR) module reduces collision rate from 79.0% (direct alignment) to 0.0%, as shown in ablation (Table 2).
• Physics-based MuJoCo stabilization closes micro-gaps caused by geometric rectification, producing physically stable resting states required for manipulation.
• Automated MLLM-driven curation pipeline scores scenes on usability, label confidence, physical attributes, and generates pick-and-place tasks with 10 spatial relation predicates.
• Top-down prioritized 6D grasp synthesis (GraspGen) paired with GPU-accelerated cuRobo motion planning produces collision-free, continuous joint-space trajectories conditioned on task instructions.
• Domain randomization with over 1700 high-resolution tabletop textures improves visual diversity and robustness of downstream visuomotor policies.

Threat model

The system assumes an honest input scenario where the adversary is not actively manipulating the data; the input is arbitrary internet imagery with no access to ground-truth scene geometry. Adversaries cannot generate physically impossible layouts that pass the automated physics verification steps. The pipeline does not consider malicious attacks or data poisoning aimed at confusing perception or manipulation stages.

Methodology — deep read

1. Threat Model & Assumptions: The pipeline assumes access to arbitrary in-the-wild internet images capturing tabletop scenes from single-view, unstructured snapshots. The adversary is not explicitly modeled; the focus is on reconstructing physically plausible scenes despite occlusion, clutter, and noisy input data. The output must be simulation-ready environments where manipulation trajectories can be executed collision-free. The system assumes standard physics via MuJoCo and does not account for adversarial manipulations.

2. Data: Raw input data consists of internet images filtered for valid tabletop surfaces. The TableVerse-100K Dataset includes 100,000 reconstructed scenes covering diverse real-world clutter, vertical stacking, and composite objects. The dataset contains nearly 1M distinct object instances spanning 35K+ semantic categories. Data splits for training/testing are not explicitly described but a 100-scene curated test set from wild images is used for evaluation.

3. Architecture/Algorithm:

• Object extraction combines Seed-1.8 open-vocabulary detection with SAM2 instance segmentation, avoiding cascading errors from large language models.
• Composite objects are deconstructed into container and contents via 3D meshes reconstructed using SAM3D. These undergo physics-guided free-fall assembly in MuJoCo to produce physically valid composite assets.
• Depth Anything 3 monocular depth estimation extracts metric point clouds for scale recovery.
• Coarse-to-fine 6-DoF registration uses rotation search + ICP alignment of segmented object point clouds and meshes.
• Layout-Consistent Collision Rectification (LCCR) uses a three-phase pipeline: hierarchical contact grouping to cluster physically interacting objects and preserve vertical stacking relations; radial graph construction to expand groups radially and eliminate lateral overlap via rigid translations preserving azimuthal spatial layout; vertical disentanglement to separate intersecting vertical meshes by raising smaller-footprint objects upwards; finally, the corrected scene undergoes a physics stabilization phase in MuJoCo to let objects settle naturally under gravity.
• Curation employs Gemini 2.5 Pro MLLM for multi-dimensional scene evaluation: plausibility gating, label error checking, physical property prediction, quality scoring, object relabeling, and procedural pick-place task generation covering 10 spatial relation predicates.
• Task-conditioned trajectory generation translates instructions into continuous joint-space robot trajectories. Candidate grasps are synthesized by the GraspGen diffusion model producing multiple 6D grasp poses per object, filtered by near-vertical downward approach constraints. Placement poses are sampled with relation constraints and verified for collision clearance. Motion planning uses GPU-accelerated cuRobo optimizing 6-phase manipulation (pre-grasp to post-place) trajectories.

4. Training Regime: The system integrates pretrained modules (Seed-1.8, SAM2/3D, Depth Anything 3, GraspGen, Gemini 2.5 Pro). No end-to-end training is reported. Hyperparameters include maximum overlap thresholds (≥50% for grouping), strict top-down grasp filtering threshold (γ_strict), and safety clearances (ϵ) for placements. No details on epochs or minibatches since components are modular or pretrained.

5. Evaluation Protocol: Metrics include Scene Collision Rate (%) measuring volumetric mesh intersections before physics relaxation, and an MLLM-generated GPT-Score combining layout fidelity (LF), visual quality (VQ), and geometry quality (GQ), each scored 1-10. Lower Average Rank across methods also tracks preference. The test set is 100 internet-derived tabletop scenes covering challenging dense clutter and various occlusion. Baselines MIDI, SAM3D, and SceneMaker are evaluated using the pipeline’s same instance masks, isolating layout generation performance. Ablations evaluate LCCR module impact by comparing direct alignment, direct+LCCR geometric rectification, and full pipeline with simulation. Qualitative visualizations show mesh interpenetration and floating artifacts removal.

6. Reproducibility: Code or dataset availability is partially implied by a project page (https://bytedance.github.io/TableVerse), but frozen weights or closed data restrictions are not stated. The use of proprietary third-party models (Gemini 2.5 Pro, Seed-1.8) may limit fully open reproduction. Details of exact hyperparameters or random seeds are not provided, leaving some reproducibility questions open.

Concrete example: Given a single internet photo depicting cluttered utensils and containers on a table, the system detects and segments each object mask, reconstructs their 3D meshes and composite relations, extracts depth point clouds, aligns and registers object poses, applies LCCR to resolve overlapping meshes while preserving layout fidelity, and solves final physics stabilization. Gemini 2.5 Pro scores the scene, verifies physicality, and generates pick-and-place instructions. GraspGen synthesizes candidate grasps satisfying top-down constraints, cuRobo plans collision-free robot trajectories to pick and place items per the instructions, producing fully interactive digital twin environments.

Technical innovations

• A deterministic Real2Sim pipeline that reconstructs dense, cluttered tabletop environments from unstructured, in-the-wild single-view RGB images with metric scale and verified physical stability, replacing prior probabilistic hallucination or procedural generation methods.
• Layout-Consistent Collision Rectification (LCCR), a hierarchical spatial optimization algorithm that disentangles mesh collisions while preserving macroscopic azimuthal layout and vertical stacking relations, enabling large-scale, physically valid scene assembly.
• Integrated physics-guided assembly of composite objects via isolated MuJoCo free-fall simulation to produce physically correct internal placements within containers.
• Automated MLLM-driven curation and annotation pipeline coupling multi-view rendering with semantic, physical, and task instruction evaluation for rigorous large-scale dataset cleansing and enrichment.
• Task-conditioned trajectory generation combining top-down prioritized 6D grasp synthesis with relation-constrained placement sampling and GPU-accelerated motion planning for collision-free pick-and-place demonstrations mapped from high-level commands.

Datasets

• TableVerse-100K — 100,000 unique tabletop environments with nearly 1 million objects and 35,000+ categories — sourced from unstructured internet images

Baselines vs proposed

• MIDI: Scene Collision Rate = 90.0% vs TableVerse: 0.0%
• SAM3D: Scene Collision Rate = 81.0% vs TableVerse: 0.0%
• SceneMaker: Scene Collision Rate = 72.0% vs TableVerse: 0.0%
• MIDI: Layout Fidelity = 5.81 vs TableVerse: 7.14
• SAM3D: Layout Fidelity = 5.73 vs TableVerse: 7.14
• SceneMaker: Layout Fidelity = 4.90 vs TableVerse: 7.14
• TableVerse ablation - Direct Alignment: Collision Rate 79.0%, Direct+LCCR: 0.0%, Direct+LCCR+Simulation: 0.0%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21017.

Fig 1: Statistical overview and scale of the TableVerse-100K dataset. Our framework establishes an unprecedented

Fig 2: Overview of the TableVerse pipeline for automated tabletop scene synthesis. Given an unstructured,

Fig 3: Physics-guided assembly workflow for composite object generation. To enrich asset diversity and interac-

Fig 4: Detailed schematic of the Layout-Consistent Collision Rectification (LCCR) module. The pipeline

Fig 5: Detailed schematic of the MLLM-driven curation and annotation pipeline. Multi-view orthographic

Fig 6: Visualization of diverse tabletop texture augmentations in TableVerse. Over 1,700 unique, high-resolution

Fig 7: Comparisons with Alternative Methods.

Fig 8: Qualitative ablation of the LCCR module. (a) Direct alignment exhibits severe, unsimulable mesh interpene-

Limitations

• Dependence on SAM3D for 3D mesh reconstruction causes failures when objects are low-resolution or occupy few pixels, leading to incorrect 3D models for nested interiors.
• Full 3D reconstruction of all scene objects via SAM3D is computationally expensive, impeding fast batch processing.
• Pipeline relies on single monocular views; no multi-view or temporal fusion to improve depth/pose accuracy.
• Use of proprietary components (e.g., Gemini 2.5 Pro MLLM) may limit open reproducibility and introduce unseen biases.
• No explicit adversarial robustness or attack simulations are performed to test security or robustness of reconstructions.
• No detailed analysis of policy learning outcomes; dataset utility for downstream RL generalization is implied but not empirically validated.

Open questions / follow-ons

• How would multi-view or video input integration improve 3D reconstruction accuracy and reduce failures on low-resolution, occluded regions?
• Can the pipeline be optimized or parallelized to speed up 3D mesh generation for large-scale batch processing?
• What are the empirical impacts of TableVerse-100K on training and generalization performance of downstream visuomotor manipulation policies under domain shifts?
• How robust is the task-conditioned trajectory generation framework to real robot uncertainties and perception noise beyond simulation?

Why it matters for bot defense

TableVerse's work on reconstructing complex, realistic, and physically valid 3D layouts from unstructured image data has parallels relevant to bot defense in how real-world scene fidelity and physical plausibility can be enforced and verified. For CAPTCHA systems involving manipulation tasks or embodied interactions, this dataset provides a valuable benchmark for training and validating generalizable policies robust to dense clutter and complex object relations typical of human environments. The LCCR method for layout collision correction can inspire approaches to detect synthetic or bot-generated layouts that fail physical consistency tests, improving bot detection accuracy. Moreover, the automated, large-scale curation pipeline driven by MLLM evaluations could be adapted to verify semantic consistency in CAPTCHA challenges dynamically. However, direct application requires domain adaptation since TableVerse focuses on robotic manipulation rather than adversarial bot activity. Still, the principles of preserving physical realism and leveraging combined perception-physics pipelines offer useful insights for enhancing CAPTCHA robustness against bot mimicry.

Cite

@article{arxiv2607_21017,
  title={ TableVerse: A Large-scale Tabletop Dataset with Real-world Grounded Layouts for Generalizable Manipulation },
  author={ Boyuan Wang and Yue Zhang and Xutao Xue and Xueyu Song and Yu Sun },
  journal={arXiv preprint arXiv:2607.21017},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21017}
}

Read the full paper

• arXiv:2607.21017 (abstract)
• arXiv:2607.21017 (PDF)