Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

Source: arXiv:2607.06516 · Published 2026-07-07 · By Songbur Wong, Xiaosong Jia, Junqi You, Bo Zhang, Pei Xu, Renqiu Xia et al.

TL;DR

This paper addresses the challenge of visually realistic, closed-loop simulation for end-to-end autonomous driving (E2E-AD). Existing simulators either support closed-loop interactivity with low visual fidelity (e.g., CARLA) or provide high-fidelity logged data with no closed-loop feedback (e.g., nuScenes). The authors propose Point as Skeleton, a generative framework that makes autoregressive driving video generation feasible in a closed-loop setting by conditioning on updated ego states, actor states, scene layouts, and a novel point-cloud skeleton representation. This skeleton decouples foreground actors from background using accumulated LiDAR point clouds and projects them to camera views as color and template-depth maps, providing stable geometric and appearance cues that reduce error accumulation over long autoregressive rollouts. They further introduce Reset-and-Roll, a novel inference method for rolling diffusion models that resets latent states between simulation steps to avoid error propagation from future-conditioned latent leakage. The system is integrated with the nuPlan simulator framework for closed-loop evaluation under ego trajectory deviations. Experimental results on nuScenes and nuPlan datasets demonstrate improved visual realism, temporal consistency, and better alignment with simulator states compared to prior generative methods, closing the gap between open-loop visual fidelity and closed-loop interactivity.

Key findings

• Point as Skeleton's autoregressive video generator achieves an FID of 5.97 and FVD of 58.3 on nuScenes, outperforming prior methods like Epona (FID 7.5, FVD 82.8) and DriveArena (FID 16.03, FVD 185.32).
• Reset-and-Roll inference eliminates future-conditioned latent states between steps, reducing error accumulation seen in full-sequence autoregressive methods, enabling stable closed-loop rollout over 100 frames.
• Point-cloud skeleton conditioning with combined color and template-depth maps reduces FVD by over 20 points compared to layout-only condition (Fig. 8) and improves projected-mask IoU by 12% versus no conditioning.
• Instance-level augmentation of foreground actors during training improves robustness in novel views, increasing projected-mask IoU from 54.52% to 58.57% on nuPlan (Table 4).
• In closed-loop evaluation with ego trajectory deviation on nuPlan-SimGen, Point as Skeleton achieves highest projected-mask IoU of 59.92%, outperforming FreeVS (54.72%) and DriveArena (46.34%), indicating better alignment of generated visuals with simulator state.
• UniAD perception metrics on nuScenes generation show improved 3D object detection mAP (+7.6 points over Epona) and segmentation (+8 points), highlighting better perception-quality observation synthesis.
• Qualitative results demonstrate robust long-horizon autoregressive generation that handles off-trajectory motion and unseen actor views better than baseline methods (Fig. 6).

Threat model

The adversary is the inherent distributional and state mismatch caused by deviations of the ego vehicle's trajectory from logged data during simulation, exposing unobserved views and actors, challenging faithful sensor observation generation. The generator must produce temporally consistent, geometrically aligned multi-view observations reacting to arbitrary planner actions. It cannot cheat by perfect future knowledge or avoid out-of-distribution visual conditions introduced by trajectory divergence.

Methodology — deep read

The authors begin by defining a closed-loop autonomous driving simulation scenario where an autoregressive video generator synthesizes camera observations at each simulation step conditioned on the current and forecasted ego and actor states and the scene layout.

1. Threat Model & Assumptions: The adversary is essentially the mismatch between simulated and real-world sensor inputs during off-log ego trajectories. The generator must handle deviations in ego motion that expose novel scene regions and actors unseen in the training data. It assumes access to logged trajectories, multi-view camera calibrations, point clouds, and HD maps.

2. Data: The experiments use nuScenes (700 train + 150 val scenes) and nuPlan-mini (64 sequences, 300 customized clips) datasets, containing multi-view images, LiDAR point clouds, 3D bounding boxes, track IDs, and HD maps. Data preprocessing involves offline accumulation of LiDAR point clouds into a global background and per-instance canonicalized foreground assets (track-indexed). Colorized points are obtained by projecting LiDAR into camera views. Foreground actors are clustered by category to build 3D point templates for template-based depth conditions.

3. Architecture: The core generator is an autoregressive video diffusion model based on Stable Diffusion SDv3.5 for video and SDv1.5 for image variants. Inputs include committed state layouts, forecasted future layouts, previous generated frames/latents, and point-cloud skeleton conditions (color and template-depth maps projected into camera views). The point skeleton decouples foreground from background to provide stable visual and geometric cues. Foreground depth conditions use category-level templates placed according to actor poses to address partial scans and sparse points. The model is trained with diffusion forcing.

4. Training: The SDv1.5 image model trains for 200 epochs on nuScenes and 150 epochs on nuPlan with batch sizes 16 and 8. The SDv3.5 video model trains for 15k iterations with batch 16 on both datasets. Instance-level augmentation corrupts 80% of foreground actor point cloud projections during training to improve robustness against viewpoint changes.

5. Evaluation Protocol: Quality metrics include FID, FVD for visual realism and temporal consistency; UniAD perception metrics (3D detection mAP, segmentation, planning); projected-mask IoU comparing generated vehicle masks with simulator vehicle masks under ego deviations (nuPlan-SimGen). Rollouts cover long horizons up to 100 frames autoregressively at 6 Hz on nuScenes, and closed-loop interactive trajectories on nuPlan. Ablations isolate effects of point skeleton modalities and augmentation. Comparisons use baseline image/video diffusers and reconstruction-generation hybrids like FreeVS and DriveArena.

6. Reproducibility: Code is openly available at https://github.com/krauwu/point-as-skeleton. Data use standard public datasets nuScenes, nuPlan-mini. Some detailed hyperparameters and augmentation protocols are provided in appendix but full training recipes not completely enumerated.

One concrete example end-to-end: at simulation step t, the planner output commits an ego pose; future waypoints form forecast layouts. These and accumulated prior frames serve as input to the video diffusion model. The point-cloud skeleton for frame t is composed by placing canonicalized actor point templates and background points per the latest state, projected as color+depth maps. Reset-and-Roll inference is applied: the latent state from denoising with only committed layouts is cached, then lookahead future layouts are used to refine the current frame denoising without committing these latents downstream. The model outputs the rendered multi-view video frame, which feeds to the driving policy for the next step decision. This process iterates, enabling closed-loop rollout with updated observations that react to off-log ego actions.

Technical innovations

• Reset-and-Roll inference for rolling diffusion enables using lookahead future layouts to improve current frame fidelity while preventing future-conditioned latent states from propagating errors to next simulation steps.
• Point Cloud Skeleton representation decomposes offline accumulated LiDAR into static background and track-indexed canonicalized foreground actor assets, reprojected as disentangled color and template-depth maps to stabilize autoregressive video synthesis.
• Template-based depth maps constructed from category-level point cloud templates provide stable geometric guidance complementary to color appearance cues, addressing partial scans and projection artifacts in foreground actors.
• nuPlan-SimGen interface enables closed-loop generative rendering directly integrated with the nuPlan traffic simulator, supporting evaluation under ego trajectory deviations with geometry-alignment metrics.

Datasets

• nuScenes — 700 train + 150 val scenes — public autonomous driving dataset with multi-view camera, LiDAR, and HD maps
• nuPlan-mini — 64 sequences (training), 300 custom clips for evaluation — open autonomous driving dataset with traffic simulation support

Baselines vs proposed

• Panacea+ [41]: FID=15.50, FVD=103.00 vs Ours (Video): FID=5.97, FVD=58.3 (nuScenes)
• MagicDrive [28]: FID=16.20, FVD=217.94 vs Ours (Video): FID=5.97, FVD=58.3
• DriveArena [37]: FID=16.03, FVD=185.32 vs Ours (Image): FID=7.09, FVD=69.47
• Epona [34]: FID=7.50, FVD=82.80 vs Ours (Video): FID=5.97, FVD=58.3
• nuPlan closed-loop: DriveArena IoU=46.34% vs FreeVS=54.72% vs Ours (Video)=59.92%
• UniAD 3DOD mAP: Epona=17.10 vs Ours (Video)=25.59; Segmentation: Epona=28.36 vs Ours (Video)=40.93

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.06516.

Fig 1: Point as Skeleton Generative Simulator. We utilize offline logs to build a foreground-

Fig 2: Reset-and-Roll Framework. Unlike full-sequence denoising [37, 20, 34], Reset-and-

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 3: Qualitative Observation. Results are generated frame-wise autoregressively on the first

Fig 4: Construction and Utilization of Point Cloud Skeleton. We separately accumulate

Limitations

• The method is evaluated on dataset trajectories and simulated deviations but not yet on completely unseen scenes or maps, limiting tests of generalization under large domain shifts.
• The foreground depth templates are category-level and may not capture fine-grained actor shape variations, possibly limiting fidelity for rare or novel actor types.
• Projected color and depth conditions rely on accumulated LiDAR point clouds which can be sparse and incomplete, causing partial reconstruction that may degrade long-horizon synthesis.
• Reset-and-Roll requires tuning of diffusion scheduler steps to cache/reuse latents correctly; improper tuning may still allow some error accumulation.
• While nuPlan-SimGen supports closed-loop ego deviations, surrounding actors mostly follow logged trajectories, limiting evaluation of full multi-agent closed-loop interactions.
• Continuous long-horizon rollout beyond 100 frames or severely off-distribution motions remain unexplored and could exhibit error growth.

Open questions / follow-ons

• How well can point-cloud skeleton conditioning scale to large scene generalization or unseen map layouts with drastically different actor configurations?
• Can the Reset-and-Roll inference framework be extended to multi-agent closed-loop simulation where all actors generate autoregressive observations conditioned on each other?
• To what extent can finer-grained or learned depth templates improve geometry grounding beyond category-level averages?
• How robust is the approach under extreme deviations in ego motion or high-density dynamic objects that cause severely incomplete point cloud projections?

Why it matters for bot defense

For bot-defense and CAPTCHA engineers working with autonomous vehicle (AV) systems, this work offers an advanced methodology for generating realistic, reactive visual sensor data in closed-loop driving simulation environments. Stable, high-fidelity visual simulation conditioned on accurate scene geometry and actor states is critical for evaluating perception and planning components under dynamic, adversarial conditions such as off-nominal trajectories or unseen environments. The Point as Skeleton approach improves autoregressive generation quality and reduces error accumulation, which could help generate more believable synthetic data or challenge test inputs that better reflect real-world AV operation at scale.

Additionally, Reset-and-Roll inference provides a novel technique to mitigate error propagation in step-wise video generation processes common in interactive systems. Insights on conditioning visual generation on accumulated point cloud skeletons might inspire conditional generation in other multi-modal adversarial settings or sensor spoofing detection frameworks. While focused on autonomous driving, the proposed generative simulation principles and systematic evaluation strategies could inform the design of robust, interactive CAPTCHA or bot detection schemes that rely on conditional, state-accurate visual synthesis under sequential feedback.

Cite

@article{arxiv2607_06516,
  title={ Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation },
  author={ Songbur Wong and Xiaosong Jia and Junqi You and Bo Zhang and Pei Xu and Renqiu Xia and Yuping Qiu and Shaofeng Zhang and Zelin Zhao and Xuechao Yan and Yuchen Zhou and Yurui Chen and Wen Guo and Hang Xu and Junchi Yan },
  journal={arXiv preprint arXiv:2607.06516},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.06516}
}

Read the full paper

• arXiv:2607.06516 (abstract)
• arXiv:2607.06516 (PDF)