GraphVid: Interactive Graph-Controllable Video Generation

Source: arXiv:2607.21580 · Published 2026-07-23 · By Vedant Shah, Onkar Susladkar, Tushar Prakash, Kiet Nguyen, Tianjio Yu, Adheesh Juvekar et al.

TL;DR

GraphVid addresses the limitations of existing controllable image-to-video (I2V) generation methods, which often rely on low-level motion cues, trajectory control, or physics-aware annotations that are cumbersome, data-intensive, and limited in modeling complex multi-object interactions. GraphVid introduces a novel graph-conditioned video generation framework that represents scene dynamics through directed interaction graphs derived from a single input image. Users can interactively edit these graphs to specify multi-object relational dynamics, which are embedded via an edge-aware graph neural network and injected as conditioning tokens into a frozen pretrained diffusion transformer backbone using parameter-efficient LoRA adaptation.

The authors also curate GraphVid-Bench, a large-scale interaction-centric video dataset with approximately 27,000 clips annotated with structured scene graphs capturing diverse physical interactions. Experiments show GraphVid achieves substantial improvements in video perceptual and reconstruction fidelity over prior trajectory-based and physics-centric methods while using two orders of magnitude less training data and substantially fewer parameters. Compared to Motion-I2V, GraphVid reduces FID by 39.9% and FVD by 37.6%, while increasing PSNR from 9.87 to 15.98 and SSIM from 0.38 to 0.61. GraphVid obtains state-of-the-art multi-object control performance with faster inference, highlighting the benefit of structured semantic interfaces and interaction-aware graph reasoning to realize physically plausible, coherent video dynamics.

Key findings

• GraphVid reduces FID by up to 39.9% (28.32 → 17.02) and FVD by 37.6% (159.32 → 99.42) compared to Motion-I2V on the MoveBench interaction-centric subset.
• PSNR improves from 9.87 to 15.98 and SSIM from 0.38 to 0.61 over Motion-I2V, indicating sharper and more structurally consistent frame reconstruction.
• GraphVid achieves lowest End-Point Error (EPE) of 2.9 among comparable-scale methods, reducing motion error by 29.3% relative to WISA (4.1) and outperforms trajectory-based methods (e.g., MagicMotion EPE 3.2).
• GraphVid uses significantly fewer parameters (0.6B trainable) and 27K training videos, compared to Wan-Move (14.5B params, 2M videos) and Motion-I2V (1.2B, 10M), yet maintains competitive or better performance.
• GraphVid attains fastest inference time (~200s) relative to other methods (Wan-Move: 1800s, WISA: 1000s), benefiting from lightweight LoRA integration and frozen large pretrained backbone.
• GraphVid-Bench dataset contains ~27K videos with an average of 7.76 nodes and 4.85 edges per graph, covering diverse interaction types such as holding, pushing, lifting, sliding, rotating, and walking.
• Multi-object interaction scenarios see consistent gains over baselines, with GraphVid improving FID to 49.45 and FVD to 291, outperforming FlashMotion and Tora on the multi-object subset of MoveBench.

Threat model

n/a — This paper is focused on controllable video generation via structured graph conditioning and does not address security, adversarial threat models, or attacker capabilities.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly modeled as this is a generative model research paper; the focus is on controllable video generation rather than security or adversarial robustness. The assumption is users provide a single conditioning image and optionally edit the interaction graph describing object relations. The model cannot violate physical interaction semantics encoded by the graph, but it does not guarantee perfect physical correctness.

2. Data: Training uses GraphVid-Bench, a newly curated large-scale dataset with ~27K annotated video clips, each 81 frames at 16fps, paired with directed interaction graphs representing entities and their relational edges (e.g., push, pull, lift). Nodes contain fused visual embeddings (from bounding box image crops via Qwen3-VL), text label embeddings, and normalized bbox coordinates; edges are open vocabulary textual descriptions encoded into embeddings. The dataset includes diverse interaction complexity: 3881 no-interaction, 10,982 single-interaction, and 12,641 multi-interaction scenes.

3. Architecture & Algorithm:

• Input image I0 is processed by a pretrained vision-language model (Qwen3-VL) for object detection and feature extraction.
• Interaction scene graph G=(V,E) is constructed where nodes V correspond to entities and edges E encode directed relational interactions with qualitative physical cues.
• Nodes: fused visual, text, and spatial features (~10,756 dimensions).
• Edges: textual descriptors encoded into 2560-dim embeddings.
• Edge-Aware Graph Reasoning module: employs a GINEConv graph isomorphism network variant that conditions message passing directly on edge attributes to propagate relational semantics through node representations.
• Node and edge features are projected to shared hidden dimension (512). The GNN outputs interaction-aware node embeddings capturing object identity and relational dynamics.
• A Graph-to-Video adapter MLP maps node embeddings (512-dim) to the frozen pretrained video diffusion transformer's latent space (4096-dim latent embedding).
• LoRA modules of rank 128 are injected into the Q, K, V, output projections of the transformer's attention layers.
• The model predicts video frames {I_t} conditioned on initial frame and edited interaction graph, learning the conditional distribution p(I_1:T | I_0, G).

4. Training Regime:

• The large LTX-Video diffusion transformer backbone and VAE encoder-decoder remain frozen.
• Only the GNN parameters, graph-to-token MLP adapter, and LoRA modules are trainable.
• Training objective uses a continuous flow-matching loss derived from the pretrained diffusion model.
• Timesteps t are sampled from a logit-normal distribution, noise added, and the model trained to denoise accordingly.
• Rank-128 LoRA adaptation reduces trainable parameters to ~0.6B.
• Training data: ~27K GraphVid-Bench video clips.

5. Evaluation Protocol:

• Metrics: FID (Frame quality), FVD (Video distribution alignment), PSNR/SSIM (reconstruction fidelity), EPE (optical flow endpoint error measuring motion accuracy).
• Main evaluation benchmark: MoveBench interaction-centric subsets (including a multi-object subtask).
• Baselines include trajectory-based and physics-condition video generation methods: Wan-Move, Motion-I2V, Tora, MagicMotion, FlashMotion, WISA.
• Ablations on model scale, training data amount, and inference speed.
• Qualitative and quantitative analyses detail interaction adherence and temporal coherence.

6. Reproducibility:

• Pretrained backbone LTX-Video used, code and dataset curation details likely provided but not stated explicitly in excerpt.
• LoRA adaptation methodology is standard, enabling reproducibility.
• Dataset GraphVid-Bench likely to be released, but not confirmed.

Concrete Example: Given a single input image containing multiple objects, GraphVid detects entities and automatically builds an initial scene graph with interactions (e.g., person holding phone; phone near payment terminal). The user can interactively edit the graph by adding or modifying edges (e.g., person leans forward, phone moves to tap terminal). The edge-aware GNN encodes this updated graph into interaction-aware embeddings, which are projected as tokens to condition the frozen diffusion transformer. The model then generates a video sequence depicting the objects moving and interacting as graph-specified, producing a temporally coherent video consistent with the edited relational dynamics.

Technical innovations

• Use of directed interaction scene graphs with rich edge attributes to model multi-object dynamics for controllable I2V generation, substituting trajectory or physics-field conditioning.
• Edge-Aware Graph Reasoning via GINEConv GNN explicitly incorporating edge semantics during message passing to obtain interaction-conditioned node embeddings.
• Lightweight graph-to-token adapter combined with LoRA modules to inject structured interaction cues into a large frozen pretrained video diffusion transformer without retraining the backbone.
• Curation of GraphVid-Bench, a large-scale (~27K) interaction-centric video dataset annotated with structured entity-relation graphs supporting relational video generation.

Datasets

• GraphVid-Bench — ~27,000 video clips — curated with structured relational annotations of multi-object interactions.

Baselines vs proposed

• Wan-Move: FID=15.56 vs GraphVid: 17.02 (slightly worse); FVD=82.17 vs 99.42; PSNR=17.21 vs 15.98; SSIM=0.61 vs 0.61
• Motion-I2V: FID=28.32 vs GraphVid: 17.02; FVD=159.32 vs 99.42; PSNR=9.87 vs 15.98; SSIM=0.38 vs 0.61
• Tora: FID=24.45 vs GraphVid: 17.02; FVD=110.47 vs 99.42; PSNR=11.27 vs 15.98; SSIM=0.54 vs 0.61
• MagicMotion: FID=26.57 vs GraphVid: 17.02; FVD=105.12 vs 99.42; PSNR=12.04 vs 15.98; SSIM=0.51 vs 0.61
• WISA: FID=25.98 vs GraphVid: 17.02; FVD=107.89 vs 99.42; PSNR=15.08 vs 15.98; SSIM=0.53 vs 0.61
• FlashMotion: FID=19.02 vs GraphVid: 17.02; FVD=104.12 vs 99.42; PSNR=14.04 vs 15.98; SSIM=0.56 vs 0.61

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21580.

Fig 1: GraphVid enables controllable multi-object image-to-video genera-

Fig 2: GraphVid Overview. From an input image, objects are detected and encoded

Fig 6: Qualitative results of GraphVid for controllable interaction-based

Fig 5: Model efficiency comparison.

Fig 5 (page 13).

Fig 6 (page 13).

Fig 7 (page 13).

Fig 8 (page 13).

Limitations

• GraphVid still underperforms substantially larger models like Wan-Move that use orders of magnitude more parameters and training data.
• No explicit adversarial robustness or evaluation under adversarial scene edits presented.
• Physical correctness is only encouraged via relational embeddings and not guaranteed by explicit physics simulation.
• GraphVid-Bench dataset curation details and code release status are not fully described, affecting reproducibility confirmation.
• Metrics focus primarily on perceptual quality and optical flow accuracy; user studies or interactive usability assessments are not reported.
• Model evaluated mostly on curated benchmarks without extensive tests on out-of-distribution real-world videos or extreme occlusions.

Open questions / follow-ons

• How well does GraphVid generalize to highly cluttered or occluded scenes where interaction graph construction becomes ambiguous or incomplete?
• Can interaction graphs be automatically inferred and edited by users in real time for interactive video editing applications?
• How might explicit physics simulators or differentiable physical modeling be incorporated to further improve physical consistency?
• Could the graph conditioning framework be extended to long-range temporal interactions beyond single video clips or to 3D scene dynamics?

Why it matters for bot defense

For bot-defense and CAPTCHA ecosystems, techniques from GraphVid highlight a promising approach to imposing semantic and relational control over generated visual content through structured interaction graphs rather than low-level motion cues or free-form text. This can improve robustness and interpretability of challenge-response tasks that rely on video or animated content by explicitly modeling multi-object physical interactions in a controllable manner.

The edge-aware graph reasoning module and graph-to-token adapter approach may inspire design of CAPTCHA schemes that require a bot to understand or generate interaction-conditioned sequences, elevating difficulty beyond simple pixel or trajectory-based manipulations. Efficient trainable adapters integrated into large frozen pre-trained backbones also suggest avenues for rapid prototyping of interaction-aware video puzzles or dynamic captchas that resist easy scripting due to their relational complexity and semantics.

Cite

@article{arxiv2607_21580,
  title={ GraphVid: Interactive Graph-Controllable Video Generation },
  author={ Vedant Shah and Onkar Susladkar and Tushar Prakash and Kiet Nguyen and Tianjio Yu and Adheesh Juvekar and Muntasir Waheed and Ismini Lourentzou },
  journal={arXiv preprint arXiv:2607.21580},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21580}
}

Read the full paper

• arXiv:2607.21580 (abstract)
• arXiv:2607.21580 (PDF)