PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing

Source: arXiv:2606.26551 · Published 2026-06-25 · By Shengbin Guo, Shaokang He, Chaoyue Meng, Shengpeng Xiao, Xunzhi Xiang, Shaofeng Zhang et al.

TL;DR

PhyEditBench addresses a critical gap in instruction-based image editing evaluation by focusing on physics-based reasoning, a capability essential for realistic edits involving physical dynamics in real-world scenes. Existing benchmarks largely overlook assessing whether models understand the physical plausibility and temporal evolution of edits governed by natural laws. PhyEditBench offers a novel multi-stage benchmark with 238 high-quality, high-resolution real-world video-derived editing instances spread across 4 major physical classes and 12 subclasses, plus 35 synthetic Anti-Physics cases designed to challenge models with counterfactual physical scenarios.

The authors conduct extensive empirical evaluation of state-of-the-art closed-source and open-source editing models, revealing substantial deficiencies in their ability to handle physics-based edits faithfully. To improve on this, they propose PhyWorld, a training-free baseline leveraging pretrained video generation models (Wan2.2) combined with a test-time scaling evolutionary search and latent reduction strategy to treat image editing as a temporal video generation problem. PhyWorld demonstrates competitive or superior performance compared to larger models, especially excelling in physically grounded reasoning and anti-physics tasks, providing evidence that video generation processes naturally internalize physical dynamics useful for editing.

The benchmark and baseline, along with a rigorous evaluation pipeline employing GPT-4o for scoring along four dimensions (consistency, instruction following, physical plausibility, and image quality), establish a new standard for measuring and advancing the physical reasoning capabilities of image editing systems.

Key findings

• PhyEditBench contains 238 real-world, high-res multi-stage editing instances plus 35 synthetic Anti-Physics cases covering 4 primary classes and 12 subclasses of physical phenomena.
• Current state-of-the-art models, including closed-source GPT-Image-1.5 and Gemini-2.5, and open-source InstructPix2Pix, FLUX.1, BAGEL, perform sub-optimally in physics-based reasoning with average overall scores < 7, frequently generating physically implausible outputs.
• PhyWorld, a training-free video-generation-based baseline with only 5B parameters, outperforms most open-source and some closed-source models on the normal subset (e.g., overall score 8.51 vs 8.47 for Seedream4.0 9B) and leads open-source models on Anti-Physics cases (score 6.39 vs 4.99 for closest open-source competitor).
• Video-based models (PhyWorld, ChronoEdit-14B) maintain temporal consistency and degrade less on long-horizon (multi-step and global) edit types (TypeD/E) compared to static editing methods, demonstrating robustness to complexity in physical state transitions.
• Physical Plausibility scoring by GPT-4o confirms substantial room for improvement; even top models score under 9 out of 10, and anti-physics tasks particularly challenge model reasoning.
• The proposed Test-Time Scaling (TTS) evolutionary optimization at inference improves output quality without additional training, reducing computational overhead compared to prior latent evolutionary search.
• The benchmark’s multi-stage format with intermediate frames allows fine-grained diagnosis of physical reasoning failures that single-final-frame benchmarks miss.
• Anti-Physics instances effectively expose reliance on memorized priors rather than genuine physical deduction, forcing models to strictly adhere to text-induced physical rules.

Threat model

The adversary is an image editing model that receives an input image and a set of edit instructions demanding physically plausible modifications consistent with real-world laws. The model may have trained on large-scale natural images/videos but is tested on whether it can perform deductive temporal reasoning reflecting physical dynamics rather than relying on memorized visual patterns or semantic correlations. It cannot cheat by ignoring physical rules in Anti-Physics scenarios or by circumventing multi-stage temporal coherence requirements.

Methodology — deep read

Threat Model & Assumptions: The primary adversary setting is an image editing model that receives a source image and multi-stage or global edit instructions, including physical constraints. The assumption is the model must reason about physical state changes and temporal dynamics without relying solely on static visual priors. Anti-Physics tests also assume adversaries attempting to hallucinate physically plausible but counterfactual results, testing true physics understanding rather than learned biases.

Data: PhyEditBench constructed from royalty-free real-world videos gathered from public stock platforms, selected for physical phenomena covering deformation/fracture, fluid dynamics, rigid body interactions, and state/environment changes. Instances are multi-stage with four temporally ordered frames (input, intermediate1, intermediate2, output) sampled from a single coherent video. Each comes with a global edit instruction, stepwise instructions, underlying physics explanations, and invariants like background held constant. There are 238 real-world sequences plus 35 synthetic Anti-Physics cases created with generative models and counterfactual prompts.

Architecture/Algorithm: PhyWorld leverages a pretrained Wan2.2 TI2V-5B video diffusion backbone that compresses inputs into latent space tokens, enabling generation of video token sequences conditioned on an input image and enhanced editing prompts. Editing is formulated as an image-to-video problem producing transition frames interpreted as implicit physical reasoning steps. Main novel components include (1) Temporal Editing Caption generation, transforming static edit instructions into detailed temporal descriptions using Qwen-3.5 vision-language model; (2) Test-Time Scaling (TTS), a single-step evolutionary latent search at the end of denoising to select optimal video samples using a pretrained Video Reward Model; (3) Latent Reduction strategy that prunes intermediate latent tokens mid-generation to reduce sequence length and computational cost without sacrificing quality.

Training Regime: PhyWorld is training-free on editing tasks, relying purely on pretrained video generation and reward models. Generation employs 121 latent frames as reasoning tokens and 30 sampling timesteps. Latent reduction is applied at timesteps 10 and 20. Multiple Gaussian noise samples initialize candidates. TTS evolutionary search is performed only once at the final denoising stage to balance quality and efficiency.

Evaluation Protocol: PhyEditBench defines five run types for per-step and global editing: step-wise input->int1 (TypeA), int1->int2 (B), int2->output (C), three-step joint from input (D), and global input->output (E). For Anti-Physics data, one edit is generated per prompt. Outputs are scored with GPT-4o, which evaluates four dimensions: Consistency (preservation of invariants and untouched content), Instruction Following (faithfulness to instructions), Physical Plausibility (alignment with physical explanations or expected phenomena), and Image Quality (visual realism). Final scores are weighted averages of these four metrics (weights: Consistency 0.2, Instruction 0.3, Physical 0.4, Image Quality 0.1). Human validation for reliability was conducted.

Reproducibility: The benchmark dataset and evaluation code are publicly available on GitHub. The PhyWorld code and pretrained Wan2.2 models (TI2V-5B) are also released. The dataset includes detailed ground-truth intermediate frames, instructions, and physics annotations. However, some internal closed-source model evaluation details may remain unavailable.

End-to-end Example: Given an input image of a glass bottle and global instruction "What happens as a hammer falls and shatters it?" PhyWorld generates edited frames corresponding to hammer impact, shard dispersion, and final shattered state. Intermediate frames correspond to physically plausible temporal states following kinetic energy transfer, momentum conservation, and Newtonian laws. Selection by video reward model chooses the best latent chain. Output is scored on consistency with background invariants, alignment with instructions, physical dynamics plausibility, and image quality, showing PhyWorld’s better adherence to gradual fracture processes compared to static editors.

Technical innovations

• Introduction of a multi-stage physics-aware image editing benchmark (PhyEditBench) with 4 physical principle classes and 12 subclasses based on real-world video sequences and detailed physics annotations.
• Design of synthetic Anti-Physics instances with counterfactual physics edit prompts to rigorously test dynamic physical reasoning instead of static memorized priors.
• Training-free image editing framework (PhyWorld) using pretrained video diffusion models as implicit physical reasoning engines through temporal editing captions and intermediate frame generation.
• Application of a single-step evolutionary Test-Time Scaling optimization with a Video Reward Model at inference to improve video generation quality efficiently without extensive computational overhead.
• Latent reduction strategy that prunes intermediate latent tokens during video generation to balance efficiency and output fidelity.

Datasets

• PhyEditBench — 238 real-world high-resolution multi-stage instances + 35 synthetic Anti-Physics cases — collected from royalty-free video stock and synthetically generated

Baselines vs proposed

• GPT-Image-1.5 (closed-source): Overall score 8.23 vs PhyWorld 8.51 on normal data
• Gemini-2.5 (closed-source): Anti-Physics score 7.07 vs PhyWorld 6.39
• Seedream4.0 (closed-source): Overall 8.47 vs PhyWorld 8.51 on normal data
• InstructPix2Pix (open-source): Overall 5.61 vs PhyWorld 8.51
• Frame2Frame (video-based open-source): Overall 6.43 vs PhyWorld 8.51
• ChronoEdit-14B (video-based open-source): Overall 8.51 vs PhyWorld 8.51 (comparable)
• BAGEL-Think (open-source): Overall 6.43 vs PhyWorld 8.51

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.26551.

Fig 1: High-quality, high-resolution, real-world examples from PhyEditBench, which

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• PhyEditBench currently limited to 238 real-world sequences plus 35 synthetic Anti-Physics samples, which may limit statistical generalization across all physical phenomena.
• Evaluation relies heavily on GPT-4o vision-language model scoring, which could introduce biases or errors despite human validation.
• PhyWorld depends on pretrained video generation models and their priors; performance may vary significantly with different or future video diffusion backbones.
• The benchmark primarily focuses on physics-aware editing in static images from video frames; applicability to full video editing tasks or broader domains is not explored.
• Some competitor models evaluated are closed-source or proprietary, limiting full reproducibility and direct comparison details.
• Anti-Physics scenarios test counterfactual reasoning but are synthetic and may not cover all physically implausible cases.

Open questions / follow-ons

• How to incorporate explicit physical simulators or physics modules into image editing models to improve physics plausibility beyond video generation priors?
• Can training video-based editing models directly on multi-stage physical reasoning tasks further improve results over training-free approaches like PhyWorld?
• How well do these physics-aware editing methods generalize to complex real-world scenes with multiple interacting objects and occlusions?
• What are the trade-offs between model scale, video generation quality, and physical reasoning accuracy in future editing architectures?

Why it matters for bot defense

For bot-defense and captcha practitioners, PhyEditBench introduces a novel axis of evaluating visual reasoning systems with a focus on physical plausibility, which could inform the design of robust visual challenge-response tests involving real-world physics. The benchmark's multi-stage and hierarchical design highlights the importance of temporal coherence and causal reasoning in sophisticated image manipulations — concepts applicable for detecting manipulative bots that fail to maintain physically consistent edits.

Furthermore, the demonstrated superiority of video-based generation and reasoning methods (as in PhyWorld) suggests that leveraging temporal and physics-aware models could help create more resilient CAPTCHAs or anti-bot visual challenges that exploit a model's inability to simulate or predict physical dynamics correctly, especially under counterfactual or adversarial conditions.

Cite

@article{arxiv2606_26551,
  title={ PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing },
  author={ Shengbin Guo and Shaokang He and Chaoyue Meng and Shengpeng Xiao and Xunzhi Xiang and Shaofeng Zhang and Qi Fan },
  journal={arXiv preprint arXiv:2606.26551},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26551}
}

Read the full paper

• arXiv:2606.26551 (abstract)
• arXiv:2606.26551 (PDF)