WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

Source: arXiv:2607.02517 · Published 2026-07-02 · By Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Qingyan Bai, Ka Leong Cheng et al.

TL;DR

WorldDirector addresses the challenge of synthesizing long-horizon, controllable video simulations with persistent dynamic object memory and viewpoint freedom, overcoming limitations of existing video world models that intertwine physical dynamics and pixel rendering. By explicitly decoupling semantic-level 3D motion planning from latent video synthesis, WorldDirector employs a large language model (LLM) to orchestrate 3D trajectories and camera paths that are projected into 2D location conditions used for video generation. It introduces an Appearance Binding mechanism that injects RGB features to anchor dynamic object appearances, preserving identity during extended occlusions and re-entries. Using a causal autoregressive chunk-based generation, it achieves long sequences with strict dynamic memory. Experiments on controlled synthetic game-based datasets demonstrate state-of-the-art performance on reconstruction fidelity and dynamic object consistency, as well as superior controllability for coordinating scene content and camera motions simultaneously.

Key findings

• WorldDirector achieves PSNR 18.127, SSIM 0.502, and LPIPS 0.359, outperforming prior state-of-the-art baselines (e.g., Yume 1.5 PSNR 14.391).
• Maintains dynamic object identity upon long-term occlusions and re-entry, verified by Dynamic Subject Consistency (DSC_DINO 0.769, DSC_CLIP 0.917), higher than baselines like HyDRA (DSC_CLIP 0.877).
• Appearance Conditioning is necessary: removing it drops PSNR from 18.127 to 16.764 and DSC_DINO from 0.769 to 0.693, causing severe identity loss.
• Explicit decoupling of semantic 3D motion planning (via LLM trajectory orchestration) from latent video synthesis enables disentangled control of object and camera motion, confirmed via qualitative comparisons.
• Causal chunk-based generation with context retrieval sustains consistent appearance and physical logic over long video sequences up to multiple minutes.
• Spatial-aware cross-attention routing effectively grounds entity-specific text captions to corresponding spatial regions, reducing semantic leakage and improving fine-grained motion control.
• Temporal Drop Mechanism on appearance condition prevents sliding artifacts while preserving identity cues for natural articulated motion.
• Ablations showed self-attention routing alone cannot replace explicit appearance conditioning without artifact degradation.

Threat model

n/a — This work is focused on controllable video world simulation with persistent dynamic memory rather than security or adversarial threat scenarios.

Methodology — deep read

1. Threat model and assumptions: The adversary is not explicitly modeled as this is a video generation framework focusing on controllability and dynamic memory rather than security threats. The system assumes access to initial images and user instructions for object and camera motions. Adversarial manipulation is outside scope.

2. Data: The authors created a synthetic game-based dataset comprising 15-second videos at 832×480 resolution and 16 fps, containing dynamic entities that disappear and reappear in camera views according to scripted trajectories. About 100 test samples were created with novel scenes. 2D bounding boxes were extracted with SAM3 for robust instance tracking across occlusions. Unique color-coded masks encode identity. Appearance references for objects were cropped from temporal context frames covering 10 seconds beyond the 5-second training window. Fine-grained captions describing entity actions were generated with Qwen2.5-VL-72B from these augmented frames.

3. Architecture/algorithm: Built on LingBot-World-Base as a foundation latent diffusion model with 3D VAE encoding inputs. Inputs include the noisy latent video chunk, concatenated with multi-modal conditioning: Location Condition (B) as color-coded 2D bounding box masks encoding spatial trajectories; Appearance Condition (A) as RGB visual anchors for objects to preserve identity; multi-granularity prompts (entity-specific captions and global video narrative); and contextual frames (M) retrieved via a dual-perspective algorithm. A Temporal Drop Mechanism sparsely retains reference frames to mitigate over-reliance on appearance features and prevent unnatural sliding. Spatial-aware weighted cross-attention routes text tokens to spatial regions to localize semantic control.

4. Training regime: Post-trained with a flow matching objective minimizing mean squared error between predicted and ground-truth velocities in latent space. Trained for 3000 steps on batches of 64 at learning rate 1e-5. Context frames are noise-free and concatenated to provide stable references. During training, the model randomly drops context frames 30% of the time to enable scratch generation.

5. Evaluation protocol: Metrics include PSNR, SSIM, LPIPS for image quality; Subject and Background Consistency from VBench for frame coherence; Dynamic Subject Consistency (DSC) computed on YOLO detected dynamic objects with DINO and CLIP similarity to evaluate identity preservation especially across occlusions. Comparisons are made against multiple prior causal autoregressive video world models, with qualitative demos scripted via the LLM. Ablation studies evaluate appearance conditioning and routing.

6. Reproducibility: Uses publicly available components where possible (SAM3, Qwen2.5-VL-72B, LingBot-World-Base), but core synthetic dataset and project code release status is unclear. Detailed prompt templates and architectural settings are described in supplemental material.

Concrete example: Given an initial frame and user instructions, the LLM plans 3D bounding box trajectories for dynamic entities and a camera trajectory. These are projected into 2D location conditioning masks. The diffusion model autoregressively generates 5-second video chunks, conditioning on appearance anchors sampled from previous chunks and retrieved context frames. During each chunk generation, temporal drop sparsifies appearance conditioning to compel the model to synthesize object motion from trajectories and text prompts rather than pixel copying. This preserves strict identity despite objects exiting and re-entering the frame, as confirmed by dynamic subject consistency metrics.

Technical innovations

• Explicit decoupling of semantic 3D motion orchestration via LLM trajectory planning from latent video synthesis, enabling physically plausible, controllable dynamic object motion independent of pixel rendering.
• Appearance Binding mechanism injecting dynamically retrieved RGB object features as visual anchors to maintain identity across prolonged out-of-view intervals.
• Temporal Drop Mechanism selectively sparsifies appearance conditioning frames to prevent sliding artifacts and encourage articulated motion synthesis.
• Spatial-aware weighted cross-attention routing aligning entity-specific textual prompts to precise bounding box regions for fine-grained multi-object semantic control during video synthesis.
• Autoregressive causal chunk-based generation integrating dual-perspective context retrieval with shifted rotary position embeddings and asymmetric attention masking to maintain spatio-temporal coherence and avoid noise contamination in context memory.

Datasets

• Synthetic game-based 15-second videos — >100 training/testing samples — Generated internally using a scripted 3D game environment with ground-truth camera and object trajectories

Baselines vs proposed

• Yume 1.5: PSNR = 14.391 vs WorldDirector: 18.127
• HY-World 1.5: PSNR = 14.782 vs WorldDirector: 18.127
• Infinite-World: PSNR = 14.574 vs WorldDirector: 18.127
• LingBot-World: PSNR = 14.116 vs WorldDirector: 18.127
• HyDRA: PSNR = 13.421 vs WorldDirector: 18.127
• WorldDirector DSC_DINO = 0.769 vs HyDRA DSC_DINO = 0.632
• WorldDirector DSC_CLIP = 0.917 vs HyDRA DSC_CLIP = 0.877
• Ablation No Appearance Condition: PSNR drops from 18.127 to 16.764, DSC_DINO drops from 0.769 to 0.693

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02517.

Fig 1: Controllable world simulation with persistent dynamic memory via WorldDirector.

Fig 2: Overview of WorldDirector. An LLM orchestrates 3D trajectories that are projected into

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Training exclusively on synthetic game-based data introduces domain gap, causing occasional unnatural locomotion and lower visual fidelity in faces.
• Evaluation limited to relatively short video segments (mostly 15 seconds) and synthetic scenarios; real-world generalization unknown.
• No adversarial robustness or security threat evaluation is conducted.
• Method depends on accurate initial object detection and bounding box tracking; errors here may degrade results.
• Temporal Drop Mechanism parameters (e.g., 16 frame dense then sparse sampling) are heuristic and may limit flexibility.
• No public release of dataset or full code at the time of writing hinders reproducibility.

Open questions / follow-ons

• How well would WorldDirector generalize to complex real-world videos beyond synthetic game environments?
• Can the model scale to longer sequences (minutes to hours) while maintaining strict dynamic identity and physical plausibility?
• How robust is the system to noisy or partial initial object detections and imperfect bounding box tracking?
• Could adversarial or unexpected user instructions cause inconsistent or implausible motion generation?

Why it matters for bot defense

WorldDirector presents a novel approach to simulate and synthesize complex video scenes with multiple dynamic objects whose movements and appearances persist consistently even when out-of-view. For bot-defense or CAPTCHA systems that aim to detect automated attacks through behavioral or motion patterns, tools like WorldDirector could be instrumental in creating realistic, controllable synthetic challenge scenarios. The explicit decoupling of semantic motion planning from rendering means that challenge designers can script precise object behaviors and trajectories, helping to test bots against sophisticated dynamic environments. However, the technique’s reliance on curated synthetic data may limit immediate applicability to real-world scenarios without further domain adaptation. Additionally, the complex conditioning mechanisms used here could inspire new CAPTCHAs that require bots to track and reason about object permanence under viewpoint changes, raising the bar for robust bot detection.

Cite

@article{arxiv2607_02517,
  title={ WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory },
  author={ Hanlin Wang and Hao Ouyang and Qiuyu Wang and Wen Wang and Qingyan Bai and Ka Leong Cheng and Yue Yu and Yixuan Li and Yihao Meng and Zichen Liu and Yanhong Zeng and Yujun Shen and Qifeng Chen },
  journal={arXiv preprint arXiv:2607.02517},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02517}
}

Read the full paper

• arXiv:2607.02517 (abstract)
• arXiv:2607.02517 (PDF)