PanoWorld: Real-World Panoramic Generation

Source: arXiv:2607.09661 · Published 2026-07-10 · By Haoyuan Li, Dizhe Zhang, Yuemei Zhou, Xiangkai Zhang, Haoran Feng, Xiaofan Lin et al.

TL;DR

PanoWorld addresses the challenge of modeling panoramic world representations with long-range memory and physical consistency across complex camera trajectories and diverse illumination. It leverages the geometric insight that panoramic equirectangular projections are rotation-equivariant, enabling the decoupling of camera rotation (treated as a known geometric transform) from translation to simplify motion modeling. The key innovations include Dense Panoramic Ray-Conditioning (DPRC) for view-dependent action modeling and Geometry-aware Memory Augmentation (GMA) to enforce spatiotemporal consistency via a memory mechanism aligned to panoramic rays. A progressive three-stage training pipeline sequentially fine-tunes the backbone for panoramic geometry, learns motion control, and augments memory for long-term coherence.

Additionally, the authors contribute World360, a large-scale panoramic video dataset with 120K clips from real UAV data and high-quality simulation incorporating multi-altitude trajectories with precise pose and depth, designed to evaluate physical consistency under challenging real-world conditions. Extensive experiments demonstrate that PanoWorld outperforms state-of-the-art baselines like Imagine360, Matrix-3D, and OmniRoam across multiple metrics measuring visual fidelity, geometric consistency, and trajectory control. The approach also enables real-time generation with minimal quality loss through an efficient autoregressive distillation. Overall, PanoWorld provides a robust framework for controllable, physically consistent panoramic video generation in complex outdoor environments.

Key findings

• PanoWorld achieves the lowest FID (27.64), FIDpole (47.21), and FIDequ (26.00) on the World360 benchmark at 480p resolution, outperforming baselines Imagine360 (FID=81.18), Matrix-3D (34.63), and OmniRoam (60.77).
• Trajectory control PSNR over temporal windows (20-25 frames) for PanoWorld is 22.83±3.73, significantly higher than Matrix-3D (20.47±2.81) and OmniRoam (18.51±4.40) at 480p.
• Geometry-aware memory augmentation (GMA) module provides +1.07 dB PSNR gain in long-term trajectory adherence compared to no-GMA baseline (Table 4), substantially reducing spatial sliding and blurring artifacts.
• World360 dataset contains 120K panoramic video sequences—70K real UAV clips and 50K AirSim360 simulations—from diverse multi-altitude trajectories with precise camera poses and depth data.
• By decoupling rotation from translation, PanoWorld avoids structural warping common in panoramic video generation, leading to better geometric consistency.
• Real-time generation via Causal Forcing achieves 161-frame panoramic video in 8 seconds on NVIDIA H20 GPU, compared to 4min 48s for full model, with negligible quality degradation.
• PanoWorld’s Dense Panoramic Ray-Conditioning models 3D translation as a dynamic light-field evolution using spherical ray unprojection, respecting 360° topology to reduce distortion compared to planar approaches.
• Pose reconstruction from generated videos using ViPE shows PanoWorld’s output tightly aligns with ground-truth camera trajectories, validating robust camera control.

Threat model

n/a — this paper focuses on panoramic video generation and world modeling rather than adversarial threat scenarios or security attacks.

Methodology — deep read

The methodology of PanoWorld centers on leveraging the rotation-equivariant property of panoramic equirectangular projections (ERPs) to simplify and improve panoramic world modeling for video generation.

1. Threat Model & Assumptions: The adversary is effectively a baseline panoramic video generator or reconstruction method. The model assumes known camera rotation can be factored out geometrically and focuses on modeling translation-induced parallax and scene dynamics. No explicit adversarial attacks are considered.

2. Data: The authors curate the World360 dataset combining 70,000 real panoramic UAV video clips with diverse trajectories across altitudes and 50,000 high-fidelity synthetic clips from the AirSim360 simulator. Data preprocessing includes three steps: Rotation Decoupling (fix headings to isolate translation), Uniform Spatial Resampling (ensures constant spatial increments across sequences), and Illumination Filtering (removes clips with poor lighting). The resulting dataset contains camera poses and depth annotations enabling geometric consistency evaluation. Training/validation splits are not explicitly detailed.

3. Architecture / Algorithm: PanoWorld builds on the Wan2.2-5B diffusion backbone, fine-tuned with LoRA to adapt to panoramic geometry. It includes two novel modules:

• Dense Panoramic Ray-Conditioning (DPRC): Maps each pixel in the ERP latent grid to a spherical ray on unit sphere, constructing a local orthonormal basis per ray to encode translation as SE(3) transformations per ray. PRoPE positional embeddings represent these rays for efficient integration into diffusion transformer blocks, enabling view-dependent intensity evolution modeling.
• Geometry-aware Memory Augmentation (GMA): A memory module that encodes past frame latents and ray pose keys into a shared PRoPE space. Confidence-guided gating fuses retrieved memory features with current diffusion features conditioned on ray attention confidence to enforce long-term spatial and radiometric consistency.

4. Training Regime: Training occurs progressively in three stages to stabilize learning:

• Stage 1: Fine-tune the backbone video model on panoramic data using a Latitude-Aware Reconstruction loss weighting by latitude angle cosine to respect ERP distortion.
• Stage 2: Freeze backbone and train DPRC module on translation-only motion data (rotation factored out) to learn view-dependent motion.
• Stage 3: Enable GMA module training to incorporate memory-based long-range coherence. No explicit epochs or batch sizes are described; hardware involves NVIDIA H20 GPUs for training and inference.

5. Evaluation Protocol: The model is benchmarked on World360 against Imagine360, Matrix-3D, and OmniRoam baselines. Metrics include FID (and variants FIDpole, FIDequ) for distribution quality, FAED for temporal consistency, NIQE, Q-Align for perceptual quality, and PSNR over 4 temporal windows (frames 20-25, 50-55, etc.) for trajectory adherence. ViPE pose reconstruction validates camera control fidelity. Ablation studies evaluate effects of memory modules and training stages. Resolution-controlled experiments are also conducted at 480p and 720p.

6. Reproducibility: The authors state models, code, and dataset will be publicly released via a project page. Detailed hyperparameters and random seed management are not explicitly given. The multi-stage training pipeline and module integration are presented clearly enough to be replicable once resources and code are available.

End-to-end example: Given an initial panoramic observation and a target camera trajectory, first rotation decoupling normalizes the heading. DPRC module encodes the translation-derived parallax as per-ray SE(3) transformations embedded with PRoPE. These condition the diffusion backbone transformer blocks for each frame in sequence. GMA retrieves relevant memories based on ray correspondence fused adaptively to enforce consistency. The trained model generates frames that follow the input trajectory with minimal drift and photometric changes.

Technical innovations

• Exploitation of rotation-equivariance in panoramic equirectangular projections to explicitly decouple rotation as a geometric transform, simplifying motion modeling to translation-induced parallax.
• Dense Panoramic Ray-Conditioning (DPRC), a per-ray SE(3)-based manifold encoding to model dynamic light-field evolution in panoramic video generation, respecting 360° topology.
• Geometry-aware Memory Augmentation (GMA), a confidence-guided memory module that retrieves and fuses historical features aligned by ray coordinates for long-range spatiotemporal and radiometric consistency.
• Three-stage progressive training pipeline separating panoramic fine-tuning, view-dependent motion learning, and memory-anchored coherence to stabilize and isolate model components effectively.

Datasets

• World360 — 120,000 panoramic video sequences — 70K real UAV captures + 50K AirSim360 simulations with multi-altitude trajectories, camera pose, and depth

Baselines vs proposed

• Imagine360: FID = 81.18 vs PanoWorld: FID = 27.64 (480p)
• Matrix-3D: FID = 34.63 vs PanoWorld: FID = 27.64 (480p)
• OmniRoam: FID = 60.77 vs PanoWorld: FID = 27.64 (480p)
• Imagine360: PSNR not available vs PanoWorld: PSNR20−25 = 22.83±3.73 (480p)
• Matrix-3D: PSNR20−25 = 20.47±2.81 vs PanoWorld: 22.83±3.73 (480p)
• OmniRoam: PSNR20−25 = 18.51±4.40 vs PanoWorld: 22.83±3.73 (480p)
• Matrix-3D: PSNR20−25 = 20.30±2.99 vs PanoWorld: 22.94±3.77 (720p)
• OmniRoam: PSNR20−25 = 18.63±3.78 vs PanoWorld: 22.94±3.77 (720p)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.09661.

Fig 1: PanoWorld is a novel framework for high-fidelity and controllable panoramic video gener-

Fig 2: The Data Curation Pipeline. The pipeline converts raw panoramic clips into high-quality

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The paper does not provide explicit adversarial robustness evaluation or security threat assessments.
• Training details such as exact epochs, batch sizes, and random seed strategies are not fully described, which may hinder strict reproducibility.
• While World360 is extensive, it focuses on UAV and aerial panoramic scenarios, limiting generalization to other panoramic video domains like indoor or street-level.
• Memory augmentation relies on confidence gating but might still struggle in extremely dynamic scenes with rapid illumination changes or novel unseen geometries.
• Performance and quality trade-offs of the real-time distilled model compared to full model are only qualitatively discussed, lacking comprehensive quantitative analysis.

Open questions / follow-ons

• How would the model perform under adversarial viewpoint perturbations or occlusions affecting memory retrieval?
• Can the rotation-equivariant principles be extended to incorporate dynamic scenes with moving objects rather than static world representations?
• What are the limits of generalization of PanoWorld when applied to non-aerial panoramic domains or indoor environments with complex lighting?
• How can the GMA memory mechanism be optimized for even longer sequences or real-time streaming videos beyond current sequence lengths?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, PanoWorld illustrates an advanced technique for modeling 360-degree panoramic environments with robust long-range spatial and temporal consistency. Its rotation-equivariant approach and geometry-informed memory augmentation provide insights into how to design video generators or simulators that maintain physical plausibility and viewpoint control over large-scale scenes. In CAPTCHA or bot-detection contexts that rely on sophisticated synthetic environment generation, PanoWorld’s techniques could inform methods for creating challenging, visually coherent panoramic scenarios that resist trivial shortcuts or attacks. Conversely, understanding PanoWorld’s memory and motion decoupling could aid in detecting generated panoramic content by analyzing inconsistencies in rotation or long-term scene persistence. Overall, the paper contributes foundational modeling advances with potential applications in evaluation and generation of complex immersive visual content in security-sensitive settings.

Cite

@article{arxiv2607_09661,
  title={ PanoWorld: Real-World Panoramic Generation },
  author={ Haoyuan Li and Dizhe Zhang and Yuemei Zhou and Xiangkai Zhang and Haoran Feng and Xiaofan Lin and Wenjie Jiang and Bo Du and Ming-Hsuan Yang and Lu Qi },
  journal={arXiv preprint arXiv:2607.09661},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.09661}
}

Read the full paper

• arXiv:2607.09661 (abstract)
• arXiv:2607.09661 (PDF)