Vera: A Layered Diffusion Model for Content-Preserving Video Editing

Source: arXiv:2606.23610 · Published 2026-06-22 · By Hongkai Zheng, Ta-Ying Cheng, Benjamin Klein, Yisong Yue, Zhuoning Yuan

TL;DR

This paper addresses a core challenge in text-driven video editing: preserving original video content outside of the edited regions. Existing video diffusion models typically regenerate entire videos, leading to unintended modifications of unchanged characters or backgrounds. Vera introduces a layered diffusion framework that generates only an edit layer and an associated alpha matte which is composited over the original video, explicitly separating edit generation from content preservation. Architecturally, Vera uses a Mixture-of-Transformers (MoT) approach with separate DiT modules generating the edit, alpha, and composite layers interacting via joint self-attention for coherent editing that respects source video dynamics.

To train Vera, the authors build a novel layered video dataset with 486K frames, combining synthetic composites, realistic single- and multi-object scenes, and videos with interactive effects like shadows and reflections, all with high-quality alpha mattes. Evaluations on background change and object addition tasks show Vera substantially outperforms recent open-source baselines on content preservation by large PSNR and SSIM margins while remaining competitive or superior on edit quality and instruction compliance. Human studies confirm preference for Vera on content faithfulness and instruction adherence. Ablations verify the importance of the layered paradigm, composite modeling, MoT architecture, and data curation in delivering state-of-the-art content-preserving video editing.

Key findings

• Vera-1.3B improves PSNR on background change by 3.5 dB and on object addition by 6.3 dB over the strongest baseline.
• Vera-14B extends these gains to +4.5 dB PSNR (background) and +7.1 dB (object addition), halving (1 - SSIM) and LPIPS perceptual error.
• Content preservation metrics (PSNR, SSIM) show large margins due to the layered design preserving untouched regions by construction.
• Vera’s alpha matte predictions are competitive with specialized video matting methods on YouTubeMatte dataset despite no dedicated matting losses.
• In human 2AFC studies over 513 trials, Vera-1.3B is significantly preferred over baselines on content preservation and instruction compliance (win rates 80–90%).
• On composition spatial and temporal quality (CS, CT), Vera performs second-best on background change and best on object addition among recent methods.
• MoT architecture with joint self-attention across three DiTs yields better content preservation and composition quality vs. single DiT baselines.
• Use of a large layered dataset with synthetic and realistic object and background changes, including visual effects, is crucial for model performance.

Threat model

The assumed adversary is a video editing system operating under user-provided text instructions that aims to generate edits without unintentionally modifying untouched regions. It does not consider malicious adversaries attempting to subvert the editing, distort preserved content, or exploit the system for stealth manipulations. The threat model focuses on typical content-preserving editing scenarios under honest usage rather than adversarial attacks or content forgery adversaries.

Methodology — deep read

The paper targets content-preserving video editing where the input source video Vsrc is modeled as a composite of preserved content Vpreserved and edit content Vedit blended by an alpha matte Aedit. The goal is to generate Vedit and Aedit given Vsrc and a text instruction C, then composite into the output video Vcomposite=(1-Aedit)◦Vpreserved + Aedit◦Vedit. In practice, Vpreserved ≈ Vsrc is approximated since semi-transparent preserved regions are minimal, and Vedit may contain semi-transparent regions (shadows, reflections).

Vera models the joint conditional distribution p(Vedit, Aedit, Vcomposite | Vsrc, C) using latent diffusion in a latent space encoded by a pretrained VAE. The model predicts velocity fields (flow matching parameterization) for all three outputs jointly via a network u_θ operating on noisy latent samples and conditioning. Three separate video transformers (DiTs), one for each output (edit, alpha, composite), form a Mixture-of-Transformers (MoT) architecture with joint self-attention over concatenated tokens to enable cross-layer interactions but allow distribution specialization per output.

All three DiTs are initialized from the same pretrained Wan2.1 text-to-video model. Additional patch embedding layers encode the conditioning source video and optional mask. Positional (RoPE) encodings and learnable layer embeddings distinguish tokens by output type. The model is trained using flow matching loss over 8,000 steps with batch size 16 on 832×480 resolution clips with 49 frames. Mask inputs are randomly dropped during training to improve robustness.

Due to lack of public layered video editing data, the authors curate a dataset with 486K frames comprising synthetic composites (from VideoMatte240K foregrounds composited on inpainted backgrounds), realistic single-object videos (from Pexels and Mixkit, processed with segmentation, matting, inpainting, captioning, and human filtering), and realistic multi-object videos with visual effects extracted via Omnimatte optimization. Each sample consists of input video, edit layer, alpha matte, and composite video.

Evaluation uses a curated benchmark of 72 object addition and 69 background change test video-prompt pairs from Pexels, DAVIS, and VACEBench covering diverse scenes and motions. Metrics include pixel-level content preservation (PSNR, SSIM) computed on preserved regions, video quality metrics (temporal flickering TF, overall consistency OC), and instruction compliance assessed via video-language models on composition spatial (CS) and temporal quality (CT) and instruction satisfaction (IS). Human 2AFC preference studies over 513 trials complement automated scores.

Baselines include seven recent open-source editing models spanning direct video-to-video translation (Ditto, Lucy-Edit, ICVE), region- and mask-conditioned editing (ReCo, VACE, VideoPainter), and layered generation (LayerFlow). All models operate on the same tasks and resolutions where applicable. Ablations isolate factors such as layered vs. end-to-end editing, MoT architecture vs. single DiT, and inclusion of different training data subsets.

One concrete example: given an input video and object addition prompt, Vera conditions on the source latent video and prompt, jointly predicts latents for edit layer, alpha matte, and composite video with the MoT transformer, then decodes the composite latent to RGB space. The alpha matte isolates edited content for compositing, avoiding modification of preserved regions. This preserves fine-grained details such as hair or shadows correctly. Evaluation on PSNR, SSIM, LPIPS confirms better preservation vs. baselines. Human raters consistently prefer Vera’s outputs for fidelity and instruction compliance.

Technical innovations

• Formulating video editing as layered RGBA video generation with explicit edit layer and alpha matte for content-preserving compositing, rather than end-to-end video regeneration.
• Extending text-to-video DiT backbone into a Mixture-of-Transformers architecture with separate DiTs for edit, alpha, and composite layers interacting through joint self-attention.
• Curating a large-scale, high-quality layered video dataset from both synthetic composites and realistic multi-object videos with visual effects including shadows and reflections.
• Jointly generating edit layer, alpha matte, and composite video latents with a unified latent diffusion model trained under flow matching loss.
• Using mask and video conditioning with learnable embeddings and positional encodings to robustly handle various editing instructions and scene complexities.

Datasets

• Layered Video Dataset — 486K frames at 832×480 resolution spanning synthetic composites and realistic single- and multi-object videos with alpha mattes — constructed by authors from VideoMatte240K, Pexels, Mixkit, and human annotations.
• VideoMatte240K — public dataset of foreground videos with high-quality alpha mattes (used for synthetic composite subset).
• Pexels and Mixkit — open-source video sources for realistic single- and multi-object scenes.

Baselines vs proposed

• Ditto: Object Addition PSNR=13.1 vs Vera-1.3B=25.3 vs Vera-14B=26.1; Background Change PSNR=21.7 vs Vera-1.3B=35.2 vs Vera-14B=36.2
• Lucy-Edit: Object Addition PSNR=19.0 vs Vera-1.3B=25.3; Background Change PSNR=22.3 vs Vera-1.3B=35.2
• ICVE: Object Addition PSNR=18.0 vs Vera-1.3B=25.3; Background Change PSNR=26.3 vs Vera-1.3B=35.2
• ReCo: Object Addition PSNR=16.8 vs Vera-1.3B=25.3; Background Change PSNR=24.9 vs Vera-1.3B=35.2
• VACE-1.3B: Object Addition PSNR=10.3 vs Vera-1.3B=25.3; Background Change PSNR=31.5 vs Vera-1.3B=35.2
• VACE-14B: Object Addition PSNR=10.1 vs Vera-14B=26.1; Background Change PSNR=31.7 vs Vera-14B=36.2
• VideoPainter: Object Addition PSNR=10.7 vs Vera-1.3B=25.3; Background Change PSNR=29.9 vs Vera-1.3B=35.2
• LayerFlow (background change only): PSNR=23.3 vs Vera-1.3B=35.2

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.23610.

Fig 1: Given an input video and a text instruction, Vera generates an edit layer together with an alpha matte that

Fig 2: Overview of the Vera inference pipeline. Given an input video and a text editing instruction, Vera’s MoT

Fig 3: The architecture of Vera compared to other video editing methods. VAE encoding, VAE decoding, and

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Approximate preserved content as identical to source video relies on minimal semi-transparent preserved regions; handling large semi-transparent preserved content remains untested.
• Evaluation focuses on two common tasks (background change and object addition), limiting generalization to other complex editing types.
• Models trained and evaluated on 832×480 resolution, potentially limiting performance on higher-resolution video editing tasks.
• No explicit adversarial robustness or security evaluation; vulnerability to adversarial prompts or video manipulations is unknown.
• Human 2AFC study sample size is modest (19 annotators, 513 trials); broader user studies on diverse content recommended.
• Layered alpha matte generation quality depends heavily on data curation pipeline; fully automated robust matting may still be challenging.

Open questions / follow-ons

• How well does the layered diffusion approach generalize to more diverse editing tasks such as object removal, relighting, or temporal edits across longer videos?
• Can the approximation of preserved content by source video be lifted to handle substantial semi-transparent preserved regions and more complex compositing scenarios?
• What are the limitations of Vera’s alpha matte accuracy in complex occlusions, fast motion, or very high-resolution videos?
• How robust is Vera to adversarial or conflicting textual instructions that might induce unwanted content modifications?

Why it matters for bot defense

Bot-defense and CAPTCHA applications rarely deal directly with video editing but can take inspiration from Vera’s layered approach to preserving unaltered content. For video or captcha generation where modification over time or frames is desired without destroying original information, layered generative models with alpha mattes offer a principled way to disentangle changes from preserved regions. This reduces risks of unintended content alteration that might otherwise confuse human raters or undermine challenge consistency.

Moreover, Vera’s Mixture-of-Transformers architecture jointly modeling multiple correlated outputs with dedicated transformer branches and cross-attention could inspire multi-channel processing in security-relevant tasks requiring careful content integrity. The emphasis on high-quality, diverse, layered training data also underlines the importance of realistic datasets for modeling detailed interactions of foreground and background, which can analogously improve robustness in synthetic media detection or bot behavior analysis.

Cite

@article{arxiv2606_23610,
  title={ Vera: A Layered Diffusion Model for Content-Preserving Video Editing },
  author={ Hongkai Zheng and Ta-Ying Cheng and Benjamin Klein and Yisong Yue and Zhuoning Yuan },
  journal={arXiv preprint arXiv:2606.23610},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.23610}
}

Read the full paper

• arXiv:2606.23610 (abstract)
• arXiv:2606.23610 (PDF)