UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Image

Source: arXiv:2606.30608 · Published 2026-06-29 · By Mohamed el amine boudjoghra, Ivan Laptev, Angela Dai

TL;DR

UnfoldArt addresses the challenging problem of reconstructing fully articulated 3D objects—including hidden interior geometries—solely from a single image or text input, without requiring supervised articulation data or multi-view inputs. Prior approaches either require extensive supervised datasets of articulated objects or multi-state/multi-view inputs, and fail to recover occluded internal structures revealed only through articulation. UnfoldArt’s key innovation is a hierarchical multi-agent system that engages in a two-round structured debate leveraging the disagreement between global and local agents and grounding this reasoning in videos generated from frozen video priors. These videos visualize the predicted motions, providing external evidence to resolve articulation ambiguity and expose previously occluded interior geometry through guided 3D latent inpainting. This agentic reasoning combined with video generative priors allows the joint recovery of semantic part decomposition, articulation parameters, and detailed geometry including interiors. Extensive experiments on in-domain and out-of-domain articulated 3D datasets demonstrate that UnfoldArt achieves state-of-the-art zero-shot performance on joint articulation accuracy and reconstructs much finer geometry than prior supervised and zero-shot baselines, illustrating strong generalization to novel object categories and complex chained kinematics. By integrating hierarchical agents with video priors in a debate framework, UnfoldArt overcomes prior limitations of supervision scarcity and incomplete geometry reconstruction from single captures.

Key findings

• UnfoldArt achieves 64.2% joint-type accuracy on the challenging Objaverse Out-of-Domain split, outperforming supervised methods Singapo (37.6%) and PhysX-Anything (18.8%), and zero-shot baselines FreeArt3D (28.6%) and Articulate-Anything (47.8%).
• On Objaverse-Household, UnfoldArt reduces joint axis error to 17.3° from 25.0° (FreeArt3D) and 23.5° (Articulate-Anything) while matching their joint-type accuracy (~57.5%).
• On the in-domain PartNet-Mobility, although Singapo yields lower reconstruction errors, UnfoldArt attains highest joint-type accuracy (77.5%) with low axis error (8.0°) and pivot error (0.15), outperforming Articulate-Anything and PhysX-Anything.
• Two-round agentic debate improves axis error from 9.29° (global-only) and 15.71° (local-only) to 8.27° with video evidence grounding (Tab. 2).
• Using video-conditioned 3D latent inpainting for interior reconstruction improves rest-state part Chamfer distance from 0.06 to 0.03 versus an alternative probe-video guided static segmentation baseline.
• Freezing the video prior and steering it with masks avoids the need for articulation-specific video data, enabling video-driven exposure of occluded geometry.
• Global-local disagreement between agents serves as a novel corrective signal for articulation parameters without requiring a trusted critic or labeled data.
• The hierarchical three-agent system (Decomposer, Grounder, Articulator) divides labor effectively across scales—semantic part motion reasoning globally, fine segmentation locally, and joint parameter arbitration.

Threat model

An adversary attempting to fool or confuse the system is limited to a single static image or text input of an articulated object without access to multi-view or multi-state observations. There is no access to ground-truth articulation data or trusted critics. The model must infer articulation parameters and occluded interiors solely from sparse observations and priors embedded in foundation models. The adversary cannot manipulate the underlying pretrained models or freeze the video generative prior.

Methodology — deep read

The threat model assumes no access to articulated supervision data or multi-view inputs and targets zero-shot 3D articulated reconstruction from a single static RGB image or text prompt. The adversary in this setting is effectively the uncertainty inherent in ambiguous articulation and occluded geometry.

Data provenance consists of three evaluation splits: PartNet-Mobility (7 classes, 77 objects, supervised training domain), Objaverse-Household (12 classes, 53 objects, partially overlapping categories with more complex parts), and Objaverse-OOD (10 classes, 40 objects, novel categories with chained kinematics). No training on articulation labels is performed; all models are zero-shot except supervised baselines.

The architecture organizes reasoning into a hierarchical multi-agent system based on foundation models. First, a TRELLIS static 3D mesh generator reconstructs the initial shape from the input. The Decomposer agent (an LLM) predicts coarse semantic part labels and motion types globally over the image. The Grounder agent produces local segmentations for each part using a segmentation model (default Gemini, with SAM3 used in ablation) applied per part. The Articulator agent predicts initial articulation parameters—joint type (revolute or prismatic), joint axis, pivot location—based on a local image crop and mesh region.

These three agents engage in a two-round global-local debate. Round 1 exploits disagreement between the Decomposer (global view) and Grounder (local view). The Articulator acts as arbiter, consolidating critiques on joint parameters and refining the initial estimate. Round 2 incorporates external motion evidence from a frozen video generative prior (WAN-VACE). A probe video simulated from the initial articulation guess provides dynamic visual evidence. Both agents judge motion plausibility and visibility of articulation. The joint parameters are updated if both agree on a revised pivot axis.

Axis refinement is performed by tracking mesh points over the probe video with optical flow, fitting Rodrigues rotation or translation motions that minimize reprojection error plus smoothness and agreement priors.

Once articulation parameters are finalized, a guided video pass conditions the frozen prior on the agreed articulation. The video shows the part articulated against the body with its interior exposed, yielding imagery revealing occluded internal geometries.

This video-conditional observation lifts to 3D through reconstruction of the articulated mesh state, labeling voxels as part, body, or ambiguous boundary via the articulation sweep volume. The ambiguous boundary shell is resolved via 3D latent-space inpainting (RePaint) in a voxel latent grid, completing hidden interiors and separating parts.

Finally, the resulting part meshes, joints, and a kinematic tree form an interactable URDF articulation asset.

Training regime: All foundation components are frozen pretrained models—no end-to-end training of the full pipeline. The debate iterates 2 rounds. Video generation produces 41 frames per part. The axis fit uses Huber loss and geometric regularizers.

Evaluation uses numerous metrics: Rest (RS) and Articulated State (AS) part bounding box IoU error (dgIoU), centroid distance (dcDist), Chamfer distance (dCD), average overlapping ratio (AOR) to detect collisions, joint axis angular error (degrees), joint pivot positional error, and joint type classification accuracy. Baselines include supervised (Singapo, PhysX-Anything) and zero-shot (FreeArt3D, Articulate-Anything) methods evaluated on held-out test splits. Ablations analyze the effect of debate rounds, segmentation model, and part reconstruction method.

An end-to-end example: given an image of a cabinet, the pipeline generates an initial TRELLIS mesh and a Flux Labs photorealistic render as input. The Decomposer predicts part labels and motions like a door. The Grounder segments the door mask locally. The Articulator proposes the door’s hinge joint parameters. The debate rounds refine these based on inconsistencies in pivot direction cues and a video showing the door opening to the left. The agreed articulation parameters condition a guided video pass revealing the previously occluded interior compartment. From the articulated mesh and video frames, the latent inpainting reconstructs clean part boundaries and interior geometry, producing a final URDF asset.

Technical innovations

• A novel two-round hierarchical agent debate framework that leverages global-local disagreement and video evidence to iteratively refine articulation parameter estimates without a trusted critic or supervised articulation data.
• Use of a frozen video generative prior both as external motion evidence grounding the agentic debate and as the means to expose occluded interior geometry through articulation-conditioned video synthesis.
• 3D latent inpainting with RePaint in voxel latent space to resolve ambiguous part-body boundaries and complete occluded interiors revealed by the articulated video frames.
• Hierarchical agent division into Decomposer (global semantic reasoning), Grounder (local segmentation), and Articulator (joint parameter prediction) to distribute articulation inference across scales effectively from single input views.
• A mask-anchored control strategy for steering frozen video diffusion models to synthesize kinematic articulation sequences consistent with predicted joint parameters.

Datasets

• PartNet-Mobility — 77 objects (7 classes) — public articulated 3D dataset
• Objaverse-Household — 53 objects (12 classes) — subset of Objaverse-animated
• Objaverse-OOD — 40 objects (10 classes, novel categories) — subset of Objaverse-animated

Baselines vs proposed

• Singapo (supervised): Joint type accuracy = 37.6% (OOD split) vs UnfoldArt: 64.2%
• PhysX-Anything (supervised): Axis error = 71.5° (OOD split) vs UnfoldArt: 25.5°
• FreeArt3D (zero-shot): Rest-state dgIoU = 0.768 (OOD split) vs UnfoldArt: 1.044 (slightly worse in dgIoU but better articulation accuracy)
• Articulate-Anything (zero-shot): Joint type accuracy = 47.8% vs UnfoldArt: 64.2% (OOD split)
• Ablation no debate rounds: Axis error = 9.29° global-only, 15.71° local-only vs Full 2-round debate: 8.0°
• Part reconstruction probe video + SegViGen: Rest-state Chamfer distance 0.06 vs Guided video + latent inpainting: 0.03

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.30608.

Fig 1: UnfoldArt generates articulated 3D objects from a single text or image input, recovering

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 2: Method overview. Given an unposed image or a text prompt as input, we generate a

Fig 3: Part reconstruction via 3D latent inpainting. The rest-pose mesh and mesh reconstructed

Fig 8 (page 4).

Limitations

• Relies on quality and fidelity of frozen video generative priors which may produce occasional failure modes or artifacts that require generation rejection.
• Segmentation performance varies with the choice of segmentation backbone; some complex chained kinematics remain challenging.
• No explicit adversarial robustness evaluation or stress testing against occlusion/lighting variations.
• Evaluation metrics focus on geometric and articulation accuracy but do not include end-user interaction or downstream task performance.
• The debate framework currently runs only two rounds; more iterative refinement could be beneficial but is unexplored.
• Interior geometry recovery depends on articulation-conditioned video quality and may struggle for very complex interiors or non-rigid parts.

Open questions / follow-ons

• How can the debate framework be extended to more rounds or broader multi-agent architectures to further improve articulation accuracy?
• Can the method be adapted to handle non-rigid or deformable articulated objects beyond revolute/prismatic joints?
• How to incorporate explicit uncertainty estimation or confidence calibration in the agent debate outputs?
• What performance does the pipeline achieve on real-world noisy or occluded inputs outside synthetic Objaverse and PartNet datasets?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, UnfoldArt’s approach highlights the potential of multi-agent reasoning coupled with video generative priors to recover complex hidden structure from sparse inputs in a zero-shot manner. This suggests new directions for CAPTCHA challenges that exploit 3D articulation understanding or motion consistency in synthesized video evidence to thwart automated Solvers. The hierarchical debate framework and leveraging global-local context disagreement can inspire mechanisms for robust bot detection through cross-validation of multi-modality cues without requiring extensive supervised training data. Furthermore, the latent inpainting approach conditioned on motion could be adapted to detect or expose synthetic manipulations that bots might use to simulate articulated 3D objects, helping improve bot detection robustness by reasoning about hidden, occluded states beyond surface appearance alone. Overall, the paper provides a roadmap for combining foundation-model priors with agentic reasoning and generative evidence for resilient zero-shot geometric understanding relevant to next-generation bot defense and interaction verification.

Cite

@article{arxiv2606_30608,
  title={ UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Image },
  author={ Mohamed el amine boudjoghra and Ivan Laptev and Angela Dai },
  journal={arXiv preprint arXiv:2606.30608},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30608}
}

Read the full paper

• arXiv:2606.30608 (abstract)
• arXiv:2606.30608 (PDF)