LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models

Source: arXiv:2607.08770 · Published 2026-07-09 · By Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang, Chin-Yang Lin, Kun-Ru Wu, Yu-Chee Tseng et al.

TL;DR

This paper addresses the challenge of recovering high-quality video from sparse, asynchronous event camera streams, a problem complicated by the ill-posed nature of reconstructing photometric intensity solely from brightness changes. Existing solutions either rely on regression models that yield blurred textures or generative models that struggle with temporal stability over long sequences. LongE2V proposes leveraging a pretrained video diffusion model backbone fine-tuned with event voxel inputs to jointly solve three tasks: event-based video reconstruction, long-horizon prediction, and frame interpolation.

Key novel mechanisms include Autoregressive Unrolling and Adaptive Context Switching to mitigate temporal drift during long sequence generation, and Reencoding Alignment with Cross Residual Correction to ensure precise bidirectional consistency in frame interpolation. Event Voxel Density Augmentation is introduced to improve robustness across varying event sensor resolutions. Extensive experiments on multiple real-world benchmarks demonstrate that LongE2V consistently outperforms state-of-the-art event-based video reconstruction, prediction, and interpolation baselines, achieving sharper textures, enhanced temporal coherence, and strong zero-shot generalization capabilities.

Overall, this paper advances event-based video analysis by integrating powerful pretrained diffusion priors with tailored architectural and training strategies, overcoming key limitations of previous approaches in terms of data efficiency, perceptual quality, and stability over long durations.

Key findings

• LongE2V achieves lower LPIPS scores than all SOTA baselines on event-based video reconstruction over ECD, MVSEC, and HQF datasets, indicating superior perceptual quality (e.g., LPIPS 0.139 vs 0.147 by next best E2VID+ on ECD).
• For event-based video prediction on the same datasets, LongE2V outperforms prior generative baseline VDM-EVFI by +4 to +6 PSNR points and reduces LPIPS by roughly 0.1, showing significantly improved long-term temporal coherence.
• In zero-shot event-based video frame interpolation (31-frame skip) on BS-ERGB and HQF, LongE2V surpasses supervised baselines like CBMNet-Large and TLXNet+ in LPIPS (0.124 vs 0.170 / 0.226 respectively), preserving fine details without training specifically for interpolation.
• Autoregressive Unrolling, Adaptive Context Switching, and inclusion of context event voxels each critically reduce error accumulation and drift during long video reconstruction, validated by ablations on HQF dataset.
• Reencoding Alignment plus Cross Residual Correction for latent space temporal flipping reduce ghosting and blur in frame interpolation, improving LPIPS by 0.037 compared to ablated variants.
• Event Voxel Density Augmentation enhances robustness against sensor resolution variations, preventing artifacts from density mismatch during inference.
• LongE2V requires only 3 cycles of autoregressive unrolling during training and maintains a 20-frame temporal context for stable video generation.
• The model successfully adapts to real-world data without fine-tuning for interpolation, demonstrating zero-shot generalization.

Methodology — deep read

The authors cast the three event-based video inverse problems—video reconstruction, prediction, and frame interpolation—as conditional video generation tasks using a pretrained video diffusion model backbone (CogVideoX). They leverage a 3D VAE that encodes video frames into latents with 4× temporal and 8× spatial compression. Events from neuromorphic sensors, represented as asynchronous streams, are discretized into voxel grids with 3 temporal bins through linear interpolation of polarity values, enabling frame-based processing aligned with video frames.

Key architectural adaptations include expanding the input projection layer of the DiT diffusion transformer to accept concatenated latents from video frames and event voxels, plus context frames and event voxels representing history. The model is fine-tuned end-to-end, fully updating the expanded first projection layer and applying LoRA adapters to DiT transformer block weights for efficient tuning. A 5% dropout on the first frame latent and optional text prompt conditioning add robustness.

Training proceeds in chunks of 49 frames at 720×480 resolution, with a 20-frame temporal context window maintained. To bridge the train-test gap caused by conditioning on ground truth frames during training but predicted frames during inference, the authors propose Autoregressive Unrolling: iteratively performing inference passes on training data to generate predictions, then using these predictions as context for subsequent fine-tuning, repeated 3 times. Adaptive Context Switching dynamically decides whether to update the context frames at chunk boundaries by measuring similarity between current and context tokens via attention maps; if below a threshold (τ=0.05), the context is updated and inference retried once to prevent drift.

For frame interpolation, the model uses both forward and backward event streams as temporal motion guidance. To correct temporal misalignment induced by non-commutative latent-space flipping in the 3D VAE, they introduce Reencoding Alignment that flips decoded pixel frames and re-encodes them before fusing. Cross Residual Correction injects residual differences between original and re-encoded latents across forward and backward branches to restore lost details and promote temporal consensus.

Event Voxel Density Augmentation randomly resizes event voxel grids during training while synchronously resizing frames to improve robustness against varying sensor resolutions and spatial densities encountered at test time.

Datasets used include the BS-ERGB real-world event dataset for training, and evaluation on subsets of ECD, MVSEC, and HQF for reconstruction and prediction, and on BS-ERGB and HQF for frame interpolation. The model is benchmarked against a range of established SOTA methods including E2VID variants, VDM-EVFI, and interpolation baselines CBMNet-Large and TLXNet+. Metrics include PSNR, SSIM, and LPIPS to assess both pixel fidelity and perceptual quality over short and long sequences. Extensive qualitative and quantitative ablations investigate all key components.

One concrete example: during inference for reconstruction on a long MVSEC sequence (~2740 frames), initial context frames are empty so the model reconstructs from zero-padded events. As frames progress, the 20-frame context is updated adaptively via attention scores. The system autoregressively generates predictions chunk by chunk while mitigating drift through corrected context switching and unrolling fine-tuning. The final output exhibits clear textures with minimal temporal artifacts even over very long durations.

The code release is mentioned on the project page, but frozen weights and detailed training scripts are not explicitly stated in the paper.

Technical innovations

• Autoregressive Unrolling: Iterative fine-tuning strategy that uses the model's own predictions as context during training to mitigate error accumulation in long-term event-based video prediction.
• Adaptive Context Switching: Dynamic mechanism using multi-head attention metrics to decide whether to update temporal context frames during inference, preventing temporal drift by retrying context updates only when relevance falls below a threshold.
• Reencoding Alignment: Method to resolve temporal misalignment in 3D VAE latent space for frame interpolation by decode-flip-encode cycles ensuring commutative consistency between latent and pixel flips.
• Cross Residual Correction: Cross injection of residuals between forward and backward reencoded latents to restore high-frequency details lost in re-encoding and promote temporal consensus during interpolation.
• Event Voxel Density Augmentation: Training-time random resizing and cropping of event voxel inputs combined with synchronous spatial augmentations of frames to enhance robustness to sensor resolution variations.

Datasets

• BS-ERGB — large-scale real-world event dataset with high-quality events and large motions — used for training and interpolation evaluation
• ECD (Event Camera Dataset) [Mueggler et al. 2017] — ∼300 frames per sequence — real-world benchmark for reconstruction and prediction
• MVSEC [Zhu et al. 2018] — long sequences up to 2,740 frames — real-world benchmark for reconstruction and prediction
• HQF [Stoffregen et al. 2020] — long sequences up to 2,430 frames — real-world benchmark for reconstruction, prediction, and interpolation

Baselines vs proposed

• Reconstruction on ECD: E2VID LPIPS=0.147 vs LongE2V LPIPS=0.139
• Reconstruction on MVSEC: HyperE2VID LPIPS=0.434 vs LongE2V LPIPS=0.405
• Prediction on HQF: VDM-EVFI PSNR=13.52 vs LongE2V PSNR=16.67; LPIPS=0.520 vs 0.229
• Interpolation on BS-ERGB (31 skip): CBMNet-Large LPIPS=0.170 vs LongE2V LPIPS=0.124
• Interpolation on HQF: CBMNet-Large PSNR=24.55 vs LongE2V PSNR=25.39; LPIPS=0.166 vs 0.105

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08770.

Fig 1: Event-based video generation. We leverage pre-trained video diffusion priors to address three distinct inverse problems within a single architecture.

Fig 2: Challenges in event-based video generation. We highlight failure

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Training performed on a single large event dataset (BS-ERGB) which may limit generalization to radically different domains or sensor types.
• Although evaluated on real-world benchmarks, robustness under extreme lighting conditions or very high-speed motions is not extensively tested.
• The computational cost and latency of the diffusion model inference in real-time applications is not discussed.
• Zero-shot interpolation performance is impressive but no fine-tuning on interpolation-specific data was performed, possibly missing further gains.
• Adaptive Context Switching relies on a threshold hyperparameter; sensitivity analysis or automatic tuning is not detailed.
• The approach depends heavily on the quality and resolution of pretrained diffusion video priors, which may limit plugging in other diffusion architectures easily.

Open questions / follow-ons

• Can the LongE2V approach generalize to other types of neuromorphic sensing modalities or event data with significantly different characteristics?
• How does the method perform under adversarial or noisy event inputs, given the reliance on precise event voxel representations?
• What are the trade-offs in latency and computational resource requirements for real-time deployment of diffusion-based event video reconstruction and prediction?
• Can the autoregressive unrolling and adaptive context mechanisms be extended to enable unsupervised continual learning or domain adaptation in dynamic scenes?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, LongE2V demonstrates an advanced approach to reconstruct high-fidelity, temporally coherent video from sparse event camera data. Event cameras' unique data structure could inspire new video generation or interpolation challenges difficult for bots to replicate, given the microsecond temporal resolution and sparse asynchronous nature of events. The integration of strong pretrained diffusion priors with temporal context adaptation presents a novel angle for generating or verifying human-interpretable video content with high naturalness and stability over long durations.

Moreover, the autoregressive unrolling and adaptive context switching techniques could inform strategies to maintain long-term coherence in challenge-response tasks requiring video generation or prediction, potentially increasing the difficulty for automated spoofing. Event voxel density augmentation also points toward robustness principles useful in handling varied sensor input quality or resolution, an aspect relevant to designing CAPTCHA tests resilient to varied attack conditions. Overall, while the system targets event-based video tasks, its architectural insights could inspire more sophisticated video synthesis or verification components in bot defense frameworks.

Cite

@article{arxiv2607_08770,
  title={ LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models },
  author={ Cheng-De Fan and Chun-Wei Tuan Mu and Chen-Wei Chang and Chin-Yang Lin and Kun-Ru Wu and Yu-Chee Tseng and Yu-Lun Liu },
  journal={arXiv preprint arXiv:2607.08770},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08770}
}

Read the full paper

• arXiv:2607.08770 (abstract)
• arXiv:2607.08770 (PDF)