Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing

Source: arXiv:2606.30599 · Published 2026-06-29 · By Sen Liang, Cong Wang, Zhentao Yu, Fengbin Guan, Zhengguang Zhou, Teng Hu et al.

TL;DR

This paper addresses the limitations in existing instruction-based video editing datasets, which largely focus on single-task appearance modifications and overlook complex structural edits and multi-task scenarios critical for practical video editing. To bridge this gap, the authors introduce Goku, a million-scale dataset comprising 2 million high-quality video editing pairs with richly annotated, instruction-aligned edits spanning basic appearance changes, structural transformations like camera and subject movement, multi-task editing, and reference-guided edits. To create these diverse and complex samples at scale, they design an automated data synthesis pipeline with progressive multi-stage filtering to ensure semantic precision, temporal coherence, and photorealism.

Leveraging the Goku dataset, they propose Goku-Edit, a dual-branch video editing model that combines a frozen multimodal large language model (MLLM) text encoder with a main appearance generation branch and an auxiliary mask prediction branch for explicit spatial control. Core novel components include RoPE-aligned cross-attention to spatially synchronize the branches and inference-time SpatialCFG guidance to enhance spatial adherence. Finally, they construct Goku-Bench, a 1,000 human-verified test case benchmark with seven specialized metrics to enable fine-grained evaluation of complex video edits. Experiments show Goku-Edit achieves up to 8% improvement over previous open-source methods on instruction following, with strong gains in physical and spatial fidelity and multi-task editing capabilities.

Key findings

• Goku dataset contains 2 million instruction-aligned video editing pairs spanning 10 core editing tasks including basic appearance edits, multi-task edits (2–5 tasks), camera movement, subject movement, and reference-based edits.
• Progressive filtering discards approximately 88% of synthesized samples during quality checks to ensure semantic precision, temporal coherence, and photorealism.
• Goku-Edit model improves instruction following by up to +8% over prior open-source video editing methods on the comprehensive Goku-Bench benchmark.
• On Goku-Bench metrics, Goku-Edit leads in Physical Rule Fidelity (PR), Spatial Relationship Accuracy (SR), Camera Motion (CM), and Subject Motion (SuM), outperforming previous methods by significant margins (e.g. PR 0.955 vs 0.694 for best prior).
• RoPE-aligned spatial cross-attention effectively resolves spatial misalignment between dual-branch inputs at different resolutions, improving structural edit control.
• SpatialCFG inference strategy amplifies spatial constraints, reducing editing spillover and boundary drifting in complex structural edits.
• Goku-Bench includes 1,000 human-verified test cases covering diverse subjects, scenes, and challenging conditions (low light, camera shake) with 7 novel editing-specific evaluation metrics.
• Goku-Edit performs noticeably better on multi-task and structural edits than baseline methods, which mostly fail at these complex tasks.

Threat model

n/a - This work is focused on dataset construction, model design, and benchmarking for instruction-based video editing and does not explicitly define or address a security adversary or threat model.

Methodology — deep read

The core of the methodology centers on three major contributions: the Goku dataset construction, the Goku-Edit model architecture, and the Goku-Bench evaluation suite.

1. Threat Model & Assumptions: The work focuses on instruction-based video editing models but does not explicitly define an adversarial threat model; rather it assumes users or developers require robust generalization to complex editing instructions including structural and multi-task changes.

2. Data Collection and Preprocessing:

• Starting with 1 million high-quality video clips from Koala-36M, clips are filtered via shot detection, aesthetic scoring, motion analysis, OCR watermark removal, and content richness screening using Gemini2.5-Pro.
• Clips are trimmed to 3–10 seconds to emphasize coherent scenes.
• Editing instructions are generated via prompts to Gemini2.5-Pro, leveraging its multimodal capabilities to identify target objects (>200 categories), generate diverse editing commands, and avoid task conflicts in multi-task edits.
• Temporal masks of target objects are extracted using Grounded-SAM2.
• Reference images and masks are perturbed for robustness against trivial copying in reference-based edits.

3. Data Generation Pipeline:

• Appearance editing tasks (Add, Remove, Swap, Alter, Style Transfer) use a combination of Minimax-Remover, VACE, Flux, and Wan2.2 models.
• Structural editing tasks (Subject Movement and Camera Movement) are decomposed into sub-problems, e.g., for subject movement, action variation is synthesized using Gemini2.5-Pro for description and Wan2.2 for conditional video generation, while position variation uses Flux for image-level repositioning and Wan2.2 for temporal coherence.
• Multi-task editing chains multiple single-task edits sequentially.
• A progressive filtering pipeline with 3 tiers (source video quality, condition verification via IoU thresholds and Gemini2.5-Pro semantic consistency, and post-synthesis validation for temporal consistency and realism) filters out 88% of samples.

4. Model Architecture - Goku-Edit:

• Dual-branch diffusion model based on Wan2.2-5B, with main branch generating edited video and an auxiliary mask branch predicting spatial masks of edited regions.
• Both branches take as inputs latent encodings of source video, target video, reference images, masks.
• Text prompts encoded by a frozen MLLM (Qwen3VL-8B).
• RoPE-aligned spatial cross-attention aligns positional embeddings across branches of different spatial resolution to avoid grid mismatch artifacts, enabling bidirectional cross-attention (mask↔video).
• SpatialCFG: an inference-time control guidance mechanism enhancing spatial mask constraints by contrasting predictions with and without cross-attention and amplifying the spatial signals separately on video and mask branches.

5. Training Regime:

• The model is fine-tuned from pretrained Wan2.2-5B weights.
• Specific epoch counts, batch sizes, hardware specs, and seeds are not detailed in the paper text.

6. Evaluation Protocol:

• Goku-Bench test set of 1,000 human-verified clips from Koala-36M with diverse content and editing instructions.
• Metrics comprised of 7 editing-specific measures: 4 general (Physical Rule Fidelity, Spatial Relationship Accuracy, Instruction Following, Overall Editing Quality) and 3 task-specific (Subject Motion, Camera Motion, Style Transfer).
• Comparison against 7 open-source and 2 commercial state-of-the-art video editing models.
• Both quantitative metrics and qualitative human evaluations (100 samples per task, multiple annotators) were conducted.

7. Reproducibility:

• The paper provides an alternative filtering pipeline using open-source models (Qwen3VL-30B) for methodology reproducibility.
• Code and weights availability are unclear or not stated explicitly.

Concrete end-to-end example (basic edit - Swap):

• From initial filtered 1M high-quality clips, Gemini2.5-Pro generates diverse Swap instructions and identifies target object/mask.
• Reference images are perturbed with lighting and pose variations.
• VACE consumes source video, mask, and reference to synthesize swapped video.
• Progressive filtering validates semantic alignment, temporal visual quality, and photorealism.
• Combined with MLLM encoding of the instruction, Goku-Edit learns to produce the edited video and mask, guided by spatial cross-attention and SpatialCFG inference.
• Evaluation performed quantitatively on Goku-Bench and qualitatively against competing baselines.

Technical innovations

• A million-scale video editing dataset 'Goku' that uniquely incorporates multi-task, structural (camera and subject movement), and reference-based edits beyond prior single-task, appearance-only datasets.
• Progressive multi-stage filtering system leveraging Gemini2.5-Pro to ensure semantic accuracy, frame-level consistency, and photorealism, removing 88% of data to guarantee quality.
• Dual-branch video editing diffusion model architecture (Goku-Edit) combining a frozen MLLM text encoder with a mask prediction branch to provide explicit spatial guidance for complex multi-faceted edits.
• RoPE-aligned spatial cross-attention mechanism that resolves positional embedding misalignment between branches operating at different spatial resolutions, enabling precise bidirectional feature fusion.
• Training-free SpatialCFG inference technique that amplifies cross-branch spatial constraints via contrasting coupled and decoupled denoising predictions, reducing boundary artifacts and enforcing spatial fidelity.

Datasets

• Goku — 2 million video editing pairs — curated from Koala-36M with filtering and synthetic editing
• Koala-36M — 36 million raw video clips — used as initial video source

Baselines vs proposed

• TokenFlow: Instruction Following(IF) = 0.42 vs Goku-Edit: 0.59
• InsV2V: IF = 0.56 vs Goku-Edit: 0.59
• StableV2V: IF = 0.45 vs Goku-Edit: 0.59
• InsViE: IF = 0.47 vs Goku-Edit: 0.59
• Omni-Video: IF = 0.43 vs Goku-Edit: 0.59
• LucyEdit: IF = 0.51 vs Goku-Edit: 0.59
• Goku-Edit achieves PR = 0.955 vs LucyEdit 0.694 and SR = 0.633 vs LucyEdit 0.598
• Goku-Edit FVD = 993.93 (lower is better) improves over InsViE 2314.89 and StableV2V 3129.58
• Commercial models Runway and Luma surpass Goku-Edit on perceptual quality (CLIP, MS, AES) but lag behind on physical plausibility (PR) and spatial reasoning (SR).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.30599.

Fig 1: Goku covers 10 core video editing task classes across basic and complex edits.

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• The dataset relies on synthetic data generation and multi-model pipelines, which may introduce subtle biases or errors despite rigorous filtering.
• The progressive filtering system discards up to 88% of data, indicating potential inefficiency and challenges in scaling or domain diversity.
• Goku-Edit’s training details (epochs, batch size, computational resources) are not fully disclosed, complicating reproducibility.
• Evaluation focuses mainly on natural, creative editing scenarios but lacks explicit adversarial or out-of-domain robustness testing.
• The benchmark size of 1,000 test cases, while specialized, remains limited for fully comprehensive real-world performance assessment.
• Reference-based editing may still permit trivial pixel copying due to the use of region-cropped reference images, despite perturbations.

Open questions / follow-ons

• How well do models trained on Goku generalize to unseen domains or real-world user-generated instructions beyond the curated dataset?
• Can the progressive filtering pipeline be adapted or optimized to reduce data discard rates while maintaining quality?
• How effectively can the dual-branch spatial control architecture handle extreme structural edits or fast-moving dynamic scenes?
• What are the performance and scaling trade-offs of the proposed RoPE-aligned cross-attention and SpatialCFG mechanisms in larger model variants or real-time applications?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, Goku and Goku-Edit represent important advances in instruction-based video editing realism and complexity, creating new challenges for detection or prevention of manipulated video content. The introduction of multi-task and structural editing greatly expands the editing attack surface compared to prior appearance-level only edits, suggesting that detection systems must evolve to understand spatial consistency, motion plausibility, and multi-modal instruction adherence.

The progressive filtering techniques and dual-branch model design insights could inspire more robust generation detection methods that exploit inconsistencies in spatial masks or instruction alignment. Additionally, the seven novel editing-specific metrics and Goku-Bench represent valuable tools for evaluating video manipulation detection robustness or for training deepfake classification models. However, the synthetic nature of the dataset and high-quality filtering raise the bar for forged video realism, potentially necessitating more sophisticated defenses incorporating temporal and physical interaction checks.

Cite

@article{arxiv2606_30599,
  title={ Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing },
  author={ Sen Liang and Cong Wang and Zhentao Yu and Fengbin Guan and Zhengguang Zhou and Teng Hu and Youliang Zhang and Yuan Zhou and Xin Li and Qinglin Lu and Zhibo Chen },
  journal={arXiv preprint arXiv:2606.30599},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30599}
}

Read the full paper

• arXiv:2606.30599 (abstract)
• arXiv:2606.30599 (PDF)