SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Source: arXiv:2606.28276 · Published 2026-06-26 · By Nadun Ranawaka, Josiah Wong, Wei-Lin Pai, Wei-Teng Chu, Tianyuan Dai, Masoud Moghani et al.

TL;DR

SimFoundry addresses the costly and labor-intensive challenge of training and evaluating robotic manipulation policies in the real world by introducing a modular, automated system for zero-shot real-to-sim scene generation from a single video input. It reconstructs interactive, sim-ready digital twins of real-world scenes, including articulated and cluttered objects, and automatically produces diverse "digital cousins"—affordance-preserving variations across objects, scene layouts, and tasks. This enables scalable policy training and evaluation in simulation with stronger generalization to new real-world conditions.

The system demonstrates high-fidelity scene reconstruction with F1 scores of 0.81–0.92 automatically and up to 0.93–0.99 with minimal user tuning. Across 7 manipulation tasks and 5 architectures, simulation evaluation results highly correlate with real-world performance (mean Pearson 0.911, mean maximum ranking violation 0.018), surpassing prior benchmarks by over 0.59 correlation. Policies trained on SimFoundry data transfer zero-shot to complex real tasks and benefit from digital cousins, improving average task success rates by 17% (object cousins), 21% (scene cousins), and 40% (task cousins). SimFoundry combines automatic pipeline design, physical simulation compatibility, and flexible data augmentation to close the sim-to-real loop effectively.

Key findings

• SimFoundry achieves zero-shot reconstruction F1 scores between 0.81 and 0.92, improving to 0.93–0.99 with 3 minutes of per-object human tuning.
• Simulation evaluations of 7 manipulation tasks and 5 policy architectures exhibit a mean Pearson correlation of 0.911 with real-world performance and a mean maximum ranking violation of 0.018.
• SimFoundry outperforms the state-of-the-art real-to-sim evaluation baseline PolaRiS by over 0.59 in Pearson correlation.
• Policies trained solely on SimFoundry synthetic data achieve up to 100% success zero-shot on some real-world tasks (e.g., Stack Dishware with DROID, Pot on Stove with YAM).
• Co-training with SimFoundry data plus limited real data yields improvements, such as increasing real-world success rates on Store Marker from 60% to 92%.
• Adding object cousins increases held-out object robustness by 50 percentage points in real-world success rates on tasks like Pot on Stove.
• Scene cousins improve layout generalization, boosting success by up to 28 points in simulation and enabling 16% success on novel cousin layouts where baseline policies get 0%.
• Task cousins enable multi-task generalization and intra-task transfer, improving simulation success by up to 60% on harder downstream tasks without target-task demonstrations.

Threat model

n/a — This work focuses on developing simulation environments for robotic policy training and evaluation and does not analyze adversarial or malicious attackers on the system.

Methodology — deep read

SimFoundry's methodology centers on creating interactive, sim-ready environments from a single real-world video, enabling policy training and evaluation with rich physical fidelity and diverse augmentations.

1. Threat Model & Assumptions: The system assumes input from a single RGB video capturing a tabletop-style scene. The adversary context is not explicit as the focus is on sim-to-real transfer and evaluation rather than security against attackers. Assumptions limit environments to planar surfaces with discrete foreground objects.

2. Data: Input is raw RGB video, optionally with depth or multiple views for background reconstruction. The dataset includes 12 reconstruction scenes and diverse manipulation tasks on two robot platforms (DROID and YAM), with evaluations spanning multi-step and articulated object interactions. Ground-truth object segmentation and task success labels come from real-world demonstrations or controlled experiments to validate simulation accuracy.

3. Architecture / Algorithm: The system pipeline has three stages:

• Extraction: A set of foundation vision models extract per-object segmentation masks, depth maps, and generate a point cloud of the scene. The foreground and background are separated through iterative segmentation and inpainting. Object poses are estimated using multi-view geometry and pose refinement models.
• Generation: Uses 2D-to-3D generation models to upsample object images into visual meshes. Articulated objects are processed via a separate articulation module that segments movable parts and infers joint parameters. Collision meshes and physics properties (mass, friction) are assigned based on scene understanding models.
• Augmentation: Generates digital cousins along three axes — object cousins vary geometry and appearance of objects; scene cousins modify spatial relationships and add distractors using semantic spatial predicates; task cousins generate new feasible goal specifications leveraging available affordances for multi-task data scaling.

The output is a physically grounded, stable simulation environment exported for downstream robot policy training and evaluation.

4. Training Regime: Policies are trained in simulation via behavior cloning or flow-matching on demonstrations generated from the reconstructed and augmented simulation environments. Training spans zero-shot sim-to-real transfer, co-training with limited real data, and multi-task learning setups. Hyperparameters, number of epochs, and hardware details are provided in supplementary materials but not exhaustively described in the main text.

5. Evaluation Protocol: Policy success is measured by end-to-end task completion rates, assessed both in simulation and on real robots across multiple tasks. Correlation metrics include Pearson correlation coefficient and mean maximum ranking violation to quantify the fidelity of simulation evaluations against real-world results. Baseline comparisons are made against PolaRiS, a leading prior real-to-sim evaluation method.

6. Reproducibility: The paper notes modular design for plugging new foundation models but does not specify open-source code or datasets. Detailed pipeline steps and component models are provided as appendices. Some underlying foundation models used are publicly available, but full reproduction may require proprietary NVIDIA simulation infrastructure.

Example end-to-end flow: From a video of a cluttered tabletop scene, SimFoundry extracts object masks and depth, generates meshes for each segmented object (e.g., drawer, cup), annotates articulation joints, reconstructs background via 3D Gaussian splatting, and assembles the complete scene in a physics simulator. Augmentation generates variants by swapping object models or rearranging layout. Policies are trained on demonstrations collected in these simulations and subsequently evaluated on the real robot to validate transfer and performance prediction.

Technical innovations

• A fully automated modular real-to-sim pipeline that reconstructs interactive sim-ready scenes including articulated and cluttered objects from a single RGB video input.
• Introduction of "digital cousins" — affordance-preserving variations along object, scene, and task axes enabling scalable synthetic data generation for improved policy generalization.
• Incorporation of multi-stage foundation vision models for object segmentation, 2D-to-3D mesh generation, articulated joint inference, and physics annotation integrated within a stable physics simulator environment.
• Sub-task level policy evaluation metrics improving real-to-sim correlation, enabling finer-grained policy benchmarking and diagnostics.

Datasets

• SimFoundry Real-to-Sim Reconstruction Scenes — 12 scenes — proprietary NVIDIA/partner real robot testbeds with RGB video captures
• Policy Evaluation Tasks — 7 manipulation tasks across DROID and YAM platforms — real-world and simulation
• Multi-task Training Tasks — 13 generated tasks plus 7 held-out real tasks for generalization — simulation and real

Baselines vs proposed

• PolaRiS baseline: mean Pearson correlation = approx. 0.32 vs SimFoundry: 0.911
• Zero-shot sim-to-real success on Pot on Stove (YAM): SimFoundry 99% vs baseline not stated
• Zero-shot sim-to-real success on Stack Dishware (DROID): SimFoundry 100% vs baseline not stated
• Store Marker real-world success: 𝜋0.5 baseline 60% vs co-trained with SimFoundry 92%
• Task cousins augmentations improve Throw Away Trash simulation success by 60% vs twin-only training

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28276.

Fig 1: Overview. SimFoundry takes a single real-world input video and automatically reconstructs an

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Relies heavily on off-the-shelf foundation models that inherit their failure modes and biases.
• Current pipeline assumes tabletop-style, planar scenes limiting generalization to multi-level or non-planar environments.
• No comprehensive adversarial evaluation or robustness testing against visual occlusions or sensor noise reported.
• SimFoundry's real-to-sim reconstruction fidelity depends on the quality of input video and may require manual tuning for best results.
• No open-source codebase or publicly shared datasets currently available, limiting reproducibility.

Open questions / follow-ons

• How to extend SimFoundry’s reconstruction and augmentation capabilities beyond planar tabletop scenes to more complex, multi-level real-world environments?
• What are the impacts of sensor noise, dynamic lighting changes, or partial occlusions on reconstruction fidelity and downstream policy transfer?
• Can the modular pipeline dynamically integrate improved foundation models over time and how does this affect overall system throughput and accuracy?
• How do different augmentation strategies impact robustness to real-world domain shifts like novel object materials or unmodeled physics?

Why it matters for bot defense

SimFoundry exemplifies an advanced approach to mitigations against physical-world data distribution shifts through scalable, controllable synthetic environment generation, a concept relevant to bot-defense practitioners concerned with simulating credible adversary environments or user behavior. The notion of generating "digital cousins" with affordance-preserving variations could inspire CAPTCHA-like challenges where interaction contexts vary semantically but preserve core functional cues, increasing attack cost. The system's strong real-to-sim policy correlation illustrates the value of leveraging high-fidelity simulations as proxies for real-world human or automated behavior testing, useful in designing evaluation frameworks for bot detection mechanisms. Practitioners in bot-defense could consider modular pipelines like SimFoundry for stress-testing behavioral policies or CAPTCHA challenges across diverse, automatically generated variants to harden models against generalization failure.

Cite

@article{arxiv2606_28276,
  title={ SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation },
  author={ Nadun Ranawaka and Josiah Wong and Wei-Lin Pai and Wei-Teng Chu and Tianyuan Dai and Masoud Moghani and Hang Yin and Yunfan Jiang and Wesley Durbano and Brandon Huynh and Yu Fang and Linxi Fan and Danfei Xu and Ruohan Zhang and Li Fei-Fei and Bowen Wen and Ajay Mandlekar and Yuke Zhu },
  journal={arXiv preprint arXiv:2606.28276},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28276}
}

Read the full paper

• arXiv:2606.28276 (abstract)
• arXiv:2606.28276 (PDF)