GS-Agent: Creating 4D Physical Worlds With Generative Simulation

Source: arXiv:2607.21522 · Published 2026-07-23 · By Hongxin Zhang, Chunru Lin, Junyan Li, Zhou Xian, Tsun-Hsuan Wang, Chuang Gan

TL;DR

GS-Agent addresses the challenging task of generating dynamic, physically realistic 4D worlds from natural language input. Unlike prior methods relying solely on large-scale generative models that often lack physical plausibility and fine control, GS-Agent implements a multi-agent framework that integrates physics engines directly into the generation loop. This design mimics human workflows for 4D content creation by decomposing the generation process into entity management (including 3D asset retrieval, material tuning, and motion control) and rendering configuration (camera and lighting). Agents collaborate, communicate via messages, and iteratively refine code executed on the physics engine to produce physically consistent, diverse, and visually coherent 4D worlds aligned to textual descriptions. Evaluated on benchmarks like NewtonGen and a novel set of complex physical scenes, GS-Agent outperforms state-of-the-art text-to-video and foundation model agents in physical invariance (State-PIS of 0.83 vs 0.57-0.62), semantic alignment, and user-rated plausibility and controllability, while providing fine-grained interactive control. By combining generative simulation with multi-agent structured reasoning, GS-Agent establishes a promising new paradigm for controllable and physically grounded 4D content creation.

Key findings

• GS-Agent achieves a State-Physical Invariance Score (State-PIS) of 0.83 on 24 NewtonGen benchmark scenes, outperforming baseline text-to-video models (Sora2: n/a, Wan2.2: n/a) and agentic baselines (SWE-Agent w/ Visual: 0.57).
• In user studies with 270 responses, GS-Agent scored highest for physical plausibility (4.33/5) and camera controllability (4.32/5) compared to baselines, indicating better perceived realism and control.
• GS-Agent's multi-agent design improves results significantly: ablation without multi-agent coordination drops State-PIS from 0.83 to 0.42 and alignment score from 29.9 to 27.9.
• The specialized Agent-Simulator interface stabilizes interaction with the physics engine; removing it reduces State-PIS from 0.83 to 0.57 and alignment from 29.9 to 26.8.
• GS-Agent supports fine-grained natural language control for motions, scene composition, camera trajectories, and rendering (e.g., slow motion, entity spawn, velocity tuning).
• GS-Agent autonomously detects and repairs simulation failures—e.g., fixing water leakage in a bathtub model by patching rigid materials—without human intervention.
• GS-Agent works robustly across different LLM backbones (GPT-5, Gemini-3-pro, Qwen3.5), with moderate performance degradation on smaller models.
• The framework produces rich multimodal 4D outputs beyond video, including depth maps, precise segmentation masks, surface normals, and particle-level dynamics suitable for downstream embodied AI.

Threat model

Not explicitly applicable; the paper focuses on generative 4D world construction rather than adversarial security. Implicit assumption is that users input honest, natural language prompts and the environment is benign. Adversarial capabilities such as manipulation or misuse are out of scope.

Methodology — deep read

1. Threat model & assumptions: The framework assumes a benevolent user providing natural language scene descriptions. The adversary model is not explicitly addressed since the paper focuses on generation quality and controllability rather than security. The system assumes access to a physics engine and pre-curated or generated 3D assets, relying on foundation models for planning and reasoning.

2. Data: GS-Agent leverages existing 3D asset libraries (BlenderKit and PolyHaven) semantically indexed by combined text and image embeddings from CLIP. For benchmarks, it uses the NewtonGen dataset (24 scenes testing 12 physical laws) and a newly collected 30-scene suite of complex multi-object interactions with dynamic cameras. Scene texts are natural language prompts describing physical behaviors and visuals. The dataset splits and preprocessing details are reported for benchmark evaluation.

3. Architecture/algorithm: GS-Agent is a multi-agent system comprising three specialized agents: Manager Agent (handles conversation, task decomposition, simulation configuration, timeline control), Entity Agent (retrieves/generates 3D assets, places entities avoiding collisions, tunes physical material parameters like Young's modulus or Poisson’s ratio using simulation feedback, controls motions via programmable commands), and Render Agent (manages camera positioning/motion, lighting setup, renders intermediate frames). The agents communicate using a SendMessage tool and generate code snippets to interact with an underlying physics engine (Genesis) through a structured API encapsulating physics solver calls and rendering controls. This agentic architecture enables iterative multi-modal feedback loops (e.g., querying simulation states, receiving rendered images/video) to refine scene generation and alignment with user text.

4. Training regime: The agents themselves leverage large language models (not strictly trained end-to-end here) such as GPT-5 for planning and code generation. Relevant hyperparameters reflect prompt engineering and interaction protocols; no supervised training for scene generation is done. The physics engine runs simulation steps with parameters configured per scene. Multiple iterations of simulation advancement and agent feedback enable convergence to physically plausible and semantically correct 4D worlds.

5. Evaluation protocol: Metrics include Physical Invariance Score (PIS) variants—Video-PIS via SAM2 segmentation from rendered frames vs the proposed State-PIS computed exactly from physics engine data (object centroids and velocities). Alignment scores between video and instruction text come from perception encoder similarities, and visual quality is quantified by the VBench aesthetics metric. Comparisons are made against top text-to-video diffusion baselines (Sora2, Wan2.2) and agentic baselines (SWE-Agent, SWE-Agent w/ visual feedback). User studies with 15 participants rate physical plausibility, camera controllability, content alignment, and aesthetics on a 5 point Likert scale. Ablations remove multi-agent design or agent-simulator interface to test importance. Additionally, robustness across different LLM backbones is evaluated.

6. Reproducibility: The framework builds on publicly available 3D asset libraries and the Genesis physics engine. The large language models GPT-5, Gemini-3-pro, and Qwen3.5 are employed, which may not be fully open-source. The codebase, detailed prompts, and tool interfaces are described, with some supplemental materials and videos on the project webpage. Exact training scripts or checkpoints of the agents are not released, limiting full reproduction.

Example flow: Given "a basketball falling and rolling over sand," the Manager Agent parses the task, delegates retrieval of a basketball model and desert assets to the Entity Agent, which fetches a mesh from BlenderKit, tunes the elastic properties to reflect the bounce and roll, and scripts motion controls. The Render Agent sets up a cinematic camera orbit and lighting conditions. Each agent checks simulation feedback and rendered frames, adjusting actions until the final rendered video matches the instructions with physically plausible dynamics and smooth camera motion.

Technical innovations

• Integration of a multi-agent system with specialized roles (Manager, Entity, Render) that collaboratively generate stable 4D worlds by iterating over code-based interaction with a physics engine.
• Agent-Simulator interface abstracts complex physics engine APIs into semantically meaningful tools, enabling autonomous agents to generate stable, physically consistent simulation code.
• Use of multimodal feedback loops including exact physics engine state queries, images, and video rendering to guide iterative refinement and self-critique during world construction.
• Automatic failure detection and correction in physics simulation (e.g., fixing water leakage by patching materials) without human intervention, demonstrating autonomous debugging within generation.
• Combining large language model reasoning for task decomposition and planning with executable physics-based generative simulation, bridging language understanding and physically grounded 4D content creation.

Datasets

• NewtonGen benchmark — 24 scenes testing 12 physical laws — public benchmark
• Complex multi-object dynamic scenes — 30 scenes collected by authors — not publicly released

Baselines vs proposed

• Sora2 (text-to-video diffusion): Video-PIS = 0.62, Alignment Score = 30.6, Aesthetic = 48.5 vs GS-Agent: Video-PIS = 0.71, Alignment Score = 32.2, Aesthetic = 47.6
• Wan2.2 (text-to-video diffusion): Video-PIS = 0.46, Alignment Score = 29.8, Aesthetic = 58.1 vs GS-Agent: Video-PIS = 0.71, Alignment Score = 32.2, Aesthetic = 47.6
• SWE-Agent (agentic baseline): Video-PIS = 0.41, State-PIS = 0.44, Alignment Score = 25.8, Aesthetic = 42.0 vs GS-Agent: Video-PIS = 0.71, State-PIS = 0.83, Alignment Score = 32.2, Aesthetic = 47.6
• SWE-Agent w/ Visual: Video-PIS = 0.49, State-PIS = 0.57, Alignment Score = 26.8, Aesthetic = 44.6 vs GS-Agent: Video-PIS = 0.71, State-PIS = 0.83, Alignment Score = 32.2, Aesthetic = 47.6
• Ablation w/o Multi-Agent: State-PIS = 0.42 vs GS-Agent 0.83
• Ablation w/o Agent-Simulator Interface: State-PIS = 0.57 vs GS-Agent 0.83

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21522.

Fig 1: GS-Agent creates 4D worlds from natural language through generative simulations,

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Dependence on current physics simulation fidelity — physics engine limitations constrain realism and scalability of generated worlds.
• Foundation models can lack perfect cinematic and motion understanding, limiting fine-grained aesthetic control and self-critique accuracy.
• No explicit adversarial evaluation or security analysis; robustness against malicious or adversarial prompts is unknown.
• Dataset for complex multi-object scenes is not publicly released, hampering reproducibility.
• Evaluation metrics rely partly on user studies and approximations; automatic metrics for physical and semantic fidelity remain imperfect.
• Large LLM backbone usage invokes cost and accessibility challenges; smaller models degrade performance noticeably.

Open questions / follow-ons

• How to extend GS-Agent to support real-time interaction or continuous editing of 4D scenes during simulation?
• Can the framework incorporate additional modalities (e.g., audio, haptics) to create richer multi-sensory physical worlds?
• How to improve automated evaluation metrics for physical consistency, cinematic quality, and semantic alignment to better guide training and assessment?
• What are effective ways to scale GS-Agent to handle extremely large or open-world 4D scenes with many interacting entities?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, GS-Agent exemplifies a sophisticated multi-agent orchestration using generated code interacting with complex physics simulation to produce grounded, consistent 4D content. This indicates a trend where attacks or automated content generation may become more physically plausible and semantically aligned, leveraging foundation models plus external engines in a closed loop. Defensive CAPTCHA systems that rely on naive synthetic video or interaction patterns may need to consider the rise of such advanced generative simulators that can create plausible physical behaviors and visuals. Moreover, GS-Agent’s autonomous error detection and correction shows how future bots might self-debug generation failures, raising the bar for detection based on mechanical failure artifacts. For practical CAPTCHA design, understanding agentic frameworks controlling physical simulation environments could inspire challenges requiring interactive physical reasoning or multi-step coordination that are currently difficult for automated attackers. In summary, GS-Agent highlights a potential future threat vector where bot-generated content gains physical and semantic sophistication through generative simulation, calling for more nuanced bot-defense approaches integrating physical reasoning and dynamic interaction analysis.

Cite

@article{arxiv2607_21522,
  title={ GS-Agent: Creating 4D Physical Worlds With Generative Simulation },
  author={ Hongxin Zhang and Chunru Lin and Junyan Li and Zhou Xian and Tsun-Hsuan Wang and Chuang Gan },
  journal={arXiv preprint arXiv:2607.21522},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21522}
}

Read the full paper

• arXiv:2607.21522 (abstract)
• arXiv:2607.21522 (PDF)