DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation
Source: arXiv:2606.31918 · Published 2026-06-30 · By Junzhe Jiang, Zipei Ma, Zijie Pan, Li Zhang
TL;DR
DriveWeaver addresses the challenge of inserting controllable, high-fidelity foreground vehicles into autonomous driving simulation videos without relying on costly or limited pre-reconstructed 3D assets. Existing methods that depend on manually curated 3D models often produce lighting and style inconsistencies, limiting scalability and realism. DriveWeaver innovates by formulating vehicle insertion as a point-cloud-conditioned video inpainting problem. By rendering pixel-level geometric conditions from vehicle LiDAR point clouds and integrating these into a latent video diffusion model via a lightweight PointAdapter module, DriveWeaver generates vehicles that are both visually realistic and geometrically consistent with the scene and trajectories. A hierarchical global-to-local generation strategy mitigates long-term temporal drift, enabling longer video sequences with stable vehicle appearance. Additionally, DriveWeaver distills the synthesized video into explicit 3D Gaussian representations, allowing real-time rendering in downstream autonomous driving simulators.
Through extensive experiments on large-scale, real-world datasets including Waymo Open Dataset and PandaSet, DriveWeaver outperforms prior state-of-the-art methods in visual fidelity metrics such as FID (e.g., 15.50 vs 49.06 for HUGSIM) and video consistency metrics like FVD. Ablation studies demonstrate the critical role of point cloud rendering and the PointAdapter module for maintaining visual and geometric quality. The distilled 3D representations from DriveWeaver can be used to efficiently render vehicles during simulation, overcoming the computational bottlenecks of per-frame diffusion. Overall, DriveWeaver provides a scalable, generalizable, and high-quality approach for controllable vehicle insertion, advancing scene editing capabilities in autonomous driving simulation.
Key findings
- DriveWeaver achieves a short-term FID of 15.50 and long-term FID of 17.50 on Waymo, outperforming HUGSIM's 49.06 and 56.99 respectively.
- Using rendered point cloud maps as pixel-level conditions improves video realism and geometric consistency compared to naive 2D point projections (FVD drops from 28.53 to 22.46).
- Adapter-based fine-tuning (PointAdapter) preserves pretrained knowledge and yields significantly better results (long-term FVD 22.46) versus full backbone finetuning alone (FVD 62.59).
- Global-to-local hierarchical inpainting strategy enables long-term video generation (up to 193 frames) without temporal drift or semantic degradation, outperforming autoregressive methods.
- Distilled 3D Gaussian representations from synthesized videos maintain strong geometric consistency with only marginal drops in visual realism (e.g., DriveWeaver-D FID 22.35 vs 15.50).
- DriveWeaver generalizes across datasets, showing consistent performance gains on PandaSet compared to baselines (short-term FID 21.54 vs HUGSIM's 56.77).
- Using classifier-free guidance (CFG) with scale 2.5 and 30 diffusion steps balances inference efficiency and quality, producing 49-frame 480p videos in ~2 minutes on NVIDIA H200 GPU.
- Sampling anchor frames at stride 4 for global anchoring supports longer sequences with stable vehicle identities and lighting.
Threat model
The adversary is the autonomous driving simulation user who controls input masked regions and vehicle point cloud trajectories to insert vehicles into scenes. The model assumes access to synchronized RGB and LiDAR sensor data but does not consider adversaries attempting to subvert or manipulate the video generation pipeline. The threats primarily concern generating realistic and geometrically consistent vehicles for evaluation, not malicious attacks.
Methodology — deep read
The core challenge tackled is controllable foreground vehicle insertion in realistic autonomous driving videos, requiring high visual realism, geometric consistency, and flexible scene editing.
Threat model and assumptions: The system assumes access to recorded autonomous driving sequences with synchronized camera images, LiDAR point clouds, and predefined vehicle trajectories. The adversary (simulator user) provides masked target insertion regions and point cloud data for vehicles to insert. It does not consider adversarial manipulations or attacks.
Data: Training data is constructed from Waymo Open Dataset and PandaSet, each providing synchronized RGB images and LiDAR point clouds at 10Hz. Approximately 10,000 training samples of vehicle instances with corresponding 3D bounding boxes and trajectories are curated. The input video resolution is 480x720 pixels over 49-frame sequences. Test sets use held-out sequences from the same datasets (~200 frames each on Waymo, ~80 frames on PandaSet).
Architecture & algorithms: DriveWeaver builds on the Wan2.1 VACE-14B latent video diffusion backbone, which includes a 3D VAE to encode videos into latent spaces, and a Diffusion Transformer (DiT) denoiser. The novel PointAdapter module is introduced as a lightweight transformer branch with fewer layers, inserted every 5 DiT blocks. It consumes concatenated RGB rendered point cloud maps, inactive background regions, and reactive inpainting masks to inject spatial geometric guidance through residual connections.
Point cloud rendering produces pixel-aligned projection maps by aggregating LiDAR points of each vehicle instance across a temporal window, then rasterizing them with fixed-radius disks to fill gaps and occlusions in the 2D image plane. These rendered conditions serve as explicit geometric constraints to the diffusion model.
Training regime: Only the PointAdapter parameters (~3B) are fine-tuned; the Wan backbone is frozen to retain pretrained priors. The training uses Adam optimizer with learning rate 1e-5, batch size 8, and linear warmup over 100 steps, training on 8 NVIDIA H200 GPUs for ~5 days (20,000 iterations). Classifier-free guidance is applied with a point condition dropping probability of 0.15. Training sequences contain 49 frames. The model is conditioned on a fixed text prompt "A realistic autonomous driving scene" to focus generation.
Inference: Generates 49-frame video clips at 480p resolution in about 2 minutes on one H200 GPU using 30 denoising steps and CFG scale 2.5. Long-term (up to 193 frames) generation uses a global-to-local hierarchical approach: sparse anchor frames are generated globally at low frame rate, then local segments are interpolated using generated anchors as dense guidance to prevent temporal drift.
Evaluation protocol: Visual realism metrics include FID, FVD, and LPIPS computed against original videos. Geometric consistency is measured by applying the OmniRe 3D Gaussian urban reconstruction pipeline to insertion results, then evaluating PSNR and SSIM of reconstructed scenes. Baselines include HUGSIM (3D mesh-based insertion), StreetCrafter (point cloud-conditioned generation), and Wan2.1 VACE-14B (masked video-to-video generation). All evaluations use sequences held out from training sets.
Reproducibility: Code is publicly released via GitHub. Model weights for PointAdapter training and datasets (Waymo, PandaSet) are publicly accessible, enabling reproduction. Some 3D reconstruction pipelines rely on established state-of-the-art methods (OmniRe). Detailed training and inference hyperparameters are provided. Dataset splits and evaluation sequences are specifically held out to ensure robust testing.
Example end-to-end: For a query video sequence with a masked vehicle region and its corresponding point cloud trajectory, the system renders rasterized point cloud maps as pixel conditions and inactive background masks. These are fed into the PointAdapter-augmented diffusion backbone to generate a completed video sequence with the inserted vehicle visually harmonized and strictly aligned with geometric constraints. The video is subsequently processed by a 3D Gaussian urban reconstruction pipeline to distill a real-time renderable 3D asset for use in simulation.
Technical innovations
- Formulating vehicle insertion as a point-cloud-conditioned latent video diffusion inpainting problem integrating dense rasterized point cloud maps as pixel-aligned geometric conditions.
- Designing a lightweight, trainable PointAdapter transformer branch that injects point cloud geometric guidance into a frozen video diffusion backbone via residual connections for efficient conditioning.
- Developing a global-to-local hierarchical inpainting generation strategy that synthesizes long video sequences (up to 193 frames) with temporal consistency by decoupling generation into sparse global anchors and local interpolations.
- Introducing a pipeline to distill synthesized video insertions into explicit 3D Gaussian representations via urban reconstruction, enabling computationally efficient real-time rendering for closed-loop autonomous driving simulation.
Datasets
- Waymo Open Dataset — 10,000 training samples with 10Hz camera and LiDAR data — public
- PandaSet — similar scale training and evaluation sequences — public
Baselines vs proposed
- HUGSIM [61]: short-term FID = 49.06, DriveWeaver: 15.50 on Waymo
- HUGSIM [61]: long-term FID = 56.99, DriveWeaver: 17.50 on Waymo
- StreetCrafter [50]: short-term FVD = 295.06, DriveWeaver: 80.17 on Waymo
- Wan2.1 VACE-14B [20]: short-term LPIPS = 0.171, DriveWeaver: 0.157 on Waymo
- HUGSIM [61]: PSNR = 36.05, DriveWeaver: 34.69 (slightly lower geometric consistency but better visual realism)
- On PandaSet, DriveWeaver achieves short-term FID 21.54 vs HUGSIM 56.77, and PSNR 31.34 vs HUGSIM 34.89
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31918.
Fig 1: Controllable high-fidelity vehicle insertion via DriveWeaver. Our
Fig 2 (page 2).
Fig 3 (page 2).
Fig 4 (page 2).
Fig 5 (page 2).
Fig 6 (page 2).
Fig 7 (page 2).
Fig 8 (page 2).
Limitations
- Despite strong geometric consistency, DriveWeaver is slightly outperformed by HUGSIM in some PSNR/SSIM metrics tied to rigid 3D mesh-based insertion.
- Long-term generation requires multi-stage inference and subsampling, increasing complexity.
- Current model requires predefined vehicle trajectories and masks; fully automatic vehicle insertion not addressed.
- Distillation into 3D Gaussian assets relies on precise 3D reconstruction pipelines and lacks depth ground truth in insertion regions.
- Computational costs remain high for training (5 days on 8 GPUs) and inference (2 min per 49-frame clip), limiting rapid iteration.
- Adversarial robustness and performance under out-of-distribution scenarios were not evaluated.
Open questions / follow-ons
- Can DriveWeaver be extended to support fully automatic scene editing with freeform vehicle placement and trajectory prediction?
- How robust is DriveWeaver under severe domain shifts such as new environments or sensor modalities not seen during training?
- Can the distillation pipeline be improved to incorporate precise depth or semantic information for better 3D asset fidelity?
- What are the tradeoffs and performance impacts when scaling to even longer sequences or higher-resolution videos?
Why it matters for bot defense
DriveWeaver introduces a novel paradigm for controllable video inpainting conditioned on sparse 3D geometry, which is directly relevant to bot-defense scenarios requiring realistic, temporally consistent foreground object insertion or manipulation in security-sensitive video streams. The approach of point cloud conditioned diffusion inpainting with temporal consistency can inspire CAPTCHA designers to adopt multi-modal conditioning signals and hierarchical generation to produce hard-to-forge, geometrically plausible video challenges for bots.
Furthermore, the distillation of generative outputs into explicit 3D assets facilitates efficient real-time rendering, a valuable consideration for large-scale CAPTCHA systems needing low-latency generation. However, DriveWeaver focuses on physical scene consistency rather than adversarial security, so additional steps would be needed to analyze robustness against attack or spoofing in CAPTCHA contexts. Overall, the paper provides important insights into scalable, controllable video generation driven by explicit geometry conditioning, which can inform advanced bot-defense multimedia challenge frameworks.
Cite
@article{arxiv2606_31918,
title={ DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation },
author={ Junzhe Jiang and Zipei Ma and Zijie Pan and Li Zhang },
journal={arXiv preprint arXiv:2606.31918},
year={ 2026 },
url={https://arxiv.org/abs/2606.31918}
}