OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing

Source: arXiv:2608.05049 · Published 2026-08-05 · By Chenxuan Miao, Yutong Feng, Yi Lu, Yunfeng Yan, Donglian Qi, Shiwei Zhang et al.

TL;DR

OmniEdit-Bench addresses the critical challenge of evaluating instruction-based video editing (IVE) models, a field where current benchmarks fall short by focusing narrowly on image-like spatial edits and lacking metrics sensitive to instruction fidelity. The paper introduces a comprehensive, multi-dimensional benchmark that captures video-specific complexities, including spatial, temporal, audio, reference-based, and reasoning-driven editing tasks. Moreover, it proposes an evaluation framework that decomposes editing quality into accuracy, preservation, realism, and consistency. A novel accuracy-aware penalty mechanism gates preservation, realism, and consistency scores by accuracy to prevent visually plausible but instruction-incorrect edits from receiving high marks. Extensive benchmarking of state-of-the-art commercial and open-source models reveals significant performance gaps, especially on temporal, audio, and reasoning tasks, illustrating that existing models excel mainly at spatial edits but struggle with multi-dimensional video editing nuances.

Key findings

• OmniEdit-Bench covers five distinct IVE tracks: spatial (240 tasks), temporal (200), audio (100), reference-based (200), and reasoning (50), enabling fine-grained diagnosis.
• Accuracy-aware penalty mechanism conditions preservation, realism, and consistency scores on accuracy, reducing inflated scores when instruction fidelity is low.
• Commercial models (e.g., KlingV3-Omni: 38.3 overall score, Wan2.7-Edit: 36.0) outperform open-source counterparts on spatial and reference tasks but not substantially on temporal or reasoning tasks.
• Temporal editing remains most challenging, with top temporal scores below 30/100 (Wan2.7-Edit at 29.0, Seedance2.0 at 25.2).
• Reasoning track performance is low across all models, with best scores below 30, indicating limited capacity for implicit instruction understanding and multi-step reasoning.
• Audio editing support is sparse and scores are low (max ~13.6 by Wan2.7-Edit) highlighting the gap in multimodal video editing capabilities.
• Vision-language model Gemini-3.1-Pro based evaluation correlates well with human judgments, with mean absolute error around 0.55–0.86 across evaluation dimensions.
• High visual preservation or temporal consistency without instruction-accurate edits leads to misleadingly high scores in previous benchmarks that OmniEdit-Bench mitigates.

Threat model

The threat model involves evaluating how well video editing models, acting as the adversaries attempting to follow user instructions, produce outputs faithfully and consistently without deviating from the specified edits. The adversary is the editing model which may exploit visual priors to produce plausible but incorrect edits. They do not have unlimited knowledge or control beyond their modeled architecture and training; the evaluation assumes no malicious intent but tests fidelity to instructions under the assumption that instruction adherence is critical.

Methodology — deep read

The paper tackles evaluation of instruction-based video editing models, recognizing that adversaries are editing systems receiving user instructions and outputs; the evaluation aims to fairly and accurately measure instruction fidelity and editing quality across video dimensions.

The authors construct OmniEdit-Bench from diverse publicly available videos, creating a benchmark with 790 total tasks across five tracks: spatial (240 instances), temporal (200), audio (100), reference-based (200), and reasoning (50), each subdivided into granular task types, e.g., camera attribute, motion semantic, human speech, and various reasoning categories (physical, causal, hypothetical).

Instructions differ in explicit (direct edit commands) and implicit (requiring user intent inference) forms, challenging models to understand complex video-specific edits. Tasks involve local to global manipulations, temporal motion patterns, audio-visual alignment, and multi-step reasoning.

To evaluate model outputs, authors deploy a four-dimension metric suite: accuracy (edit fidelity to instruction), preservation (unchanged content fidelity), realism (perceptual plausibility including physical and temporal aspects), and consistency (temporal/spatial/audio coherence). Each score ranges from 1 to 5, assessed automatically using the Gemini-3.1-Pro vision-language model with custom prompts tuned for each track and evaluation dimension.

A key novelty is the accuracy-aware penalty: accuracy score (A) normalized to Aˆ = A/5 is used to modulate other metrics P, R, C via multiplication (P'=AˆP, R'=AˆR, C'=AˆC) before computing final weighted score = 0.5A + 0.2P' + 0.15R' + 0.15C'. This gating reduces inflated scores from visually plausible but incorrect edits, enforcing instruction fidelity.

The authors evaluated nine state-of-the-art models, covering open-source (Ditto, UniVideo, VIVA) and commercial (KlingV3-Omni, Seedance2.0, Grok Imagine, Wan2.7-Edit) systems, processing videos mostly at 480P or 720P resolutions. Spatial tasks dominate training data granularity, with temporal and audio fewer but more complex.

No explicit training protocol is introduced as the paper focuses on benchmarking; models are evaluated on held-out test sets with extensive human annotations for validation. Human studies confirm strong correlation between Gemini-3.1-Pro automated scores and human judgments, indicating reproducibility and reliability.

One example: for a temporal editing task requiring a camera movement along an arc left, models generate edited videos subjected to Gemini-3.1-Pro prompts analyzing accuracy of camera parameter changes, preservation of irrelevant content, realism of changes and motion, and temporal consistency. The final score reflects the gating effect of accuracy on other metrics.

The benchmark and evaluation code are publicly released for reproducibility. Some tested commercial models have partially closed components, but evaluation dataset and framework remain open for future work.

Overall, the methodology rigorously decomposes IVE evaluation into multi-track, multi-dimension metrics emphasizing instruction satisfaction and realistic video-specific challenges not captured by prior image-centric benchmarks.

Technical innovations

• Systematic decomposition of instruction-based video editing tasks into five multi-dimensional tracks (spatial, temporal, audio, reference-based, reasoning) with explicit/implicit instruction categorization.
• A novel accuracy-aware penalty mechanism that gates preservation, realism, and consistency scores with accuracy to strictly enforce instruction fidelity in evaluation.
• Automated multi-dimensional evaluation using tailored vision-language model Gemini-3.1-Pro prompts for fine-grained assessment of diverse editing aspects.
• Inclusion of audio and high-level reasoning editing tasks, rarely addressed in prior video editing benchmarks, enabling comprehensive multi-modal and cognitive evaluation.
• Unified benchmark integrating fragmented prior evaluation domains into a single comprehensive and structured dataset and scoring framework.

Datasets

• OmniEdit-Bench — 790 instruction-based video editing tasks spanning multiple dimensions — Publicly released by authors.

Baselines vs proposed

• Runway Aleph: Overall Score = 24.2 vs KlingV3-Omni: 38.3
• Grok Imagine: Overall Score = 19.0 vs Wan2.7-Edit: 36.0
• Seedance2.0*: Temporal Track Score = 25.2 vs Wan2.7-Edit: 29.0
• Ditto: Spatial Track = 22.9 vs UniVideo: 39.9
• KlingV3-Omni Reference Track = 49.9 vs Grok Imagine Reference Track = 13.1

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05049.

Fig 1: Overview of our evaluation tracks. This figure illustrates representative tasks from

Fig 2: Taxonomy of video editing tasks. We classify the benchmark into five tracks: Spatial

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Benchmark video resolution is limited to 480P/720P to balance scale and compute, not reflecting high-res real-world production editing scenarios.
• Evaluation relies heavily on Gemini-3.1-Pro VLM automated scores; while validated with human annotations, VLM may miss subtle perceptual or semantic errors.
• Current models evaluated mostly predate full audio editing capabilities, resulting in sparse and low audio track performance.
• Reasoning track tasks are relatively few (50 total), limiting statistical power for complex multi-step inference evaluation.
• Proprietary models tested have closed training data and architecture details, complicating precise attribution of performance gaps.
• No adversarial or robustness testing against intentionally misleading instructions or out-of-distribution content is performed.

Open questions / follow-ons

• How can future models better integrate temporal and audio coherence while maintaining accurate instruction execution in complex dynamic scenes?
• What architectural or training approaches improve implicit instruction reasoning and multi-step causal editing beyond current capabilities?
• How to further validate and improve automated VLM evaluation metrics to capture subtle or high-level semantic correctness and naturalness?
• Can adversarial or robustness benchmarking of IVE models reveal failure modes masked by current evaluation frameworks?

Why it matters for bot defense

OmniEdit-Bench provides bot-defense and CAPTCHA engineers with a detailed framework to evaluate video content manipulation fidelity in instruction-driven contexts, illustrating challenges in ensuring edits truly reflect user commands rather than superficially plausible alterations. For CAPTCHA, where verifying human instructions and semantic correctness is critical, OmniEdit-Bench's emphasis on accuracy-aware scoring and multi-dimensional evaluation offers insights into developing robust detection of deepfake or automated video editing attempts that may evade superficial quality tests. The inclusion of audio and reasoning tracks highlights that attackers might exploit non-spatial dimensions for subtle manipulations, underscoring the importance of multi-modal and semantic consistency checks. Practitioners can apply similar accuracy-gated evaluation techniques or VLM-based proxies for scalable detection and quality assurance in video-based bot-defense systems. The benchmark also sets out key gaps current models face, guiding defensive strategies that focus on model weaknesses, e.g., temporal or reasoning-based inconsistencies.

Cite

@article{arxiv2608_05049,
  title={ OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing },
  author={ Chenxuan Miao and Yutong Feng and Yi Lu and Yunfeng Yan and Donglian Qi and Shiwei Zhang and Yu Liu and Xi Chen and Hengshuang Zhao },
  journal={arXiv preprint arXiv:2608.05049},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05049}
}

Read the full paper

• arXiv:2608.05049 (abstract)
• arXiv:2608.05049 (PDF)