HumanForge: A Human-Centric Deepfake Video Benchmark with Multi-Agent Forgery Rationales

Source: arXiv:2607.08705 · Published 2026-07-09 · By Wenbo Xu, Zhimin Chen, Xiaojie Liang, Hengrui Liu, Wei Lu

TL;DR

HumanForge introduces a comprehensive human-centric deepfake video benchmark designed to address critical limitations in existing datasets, such as narrow scenario coverage and lack of multi-modal alignment. Unlike prior benchmarks focusing on face-swapping or isolated subjects, HumanForge covers four core generative video scenarios: audio-driven lip-synchronization, pose-driven motion transfer, complex human-human and human-object interactions, and semantic-driven text-to-video editing. To automatically label the dataset with fine-grained, context-aware forgery rationales, the authors propose Gen2Anno, a novel multi-agent annotation pipeline that contrasts the generation expected state—derived from prompt metadata and references—against the actual forged video state. This contrastive verification produces structured omnibus annotations that include binary authenticity, artifact categories, spatial-temporal localization, and natural-language reasoning.

HumanForge contains over 18,000 videos synthesized from more than ten state-of-the-art diffusion-based video generators and editors, leveraging about 2,000 real source videos from diverse datasets such as HDTF, FF++, FFIW, and TikTok. Extensive evaluation benchmarking existing deepfake detectors and large multimodal models shows HumanForge is a challenging dataset with poor zero-shot generalization and limited fine-grained reasoning performance. The release of HumanForge and Gen2Anno code offers a significant new foundation for advancing robust, interpretable human-centric video forgery detection beyond existing image or isolated-video datasets.

Key findings

• HumanForge dataset contains 18,849 synthetic videos across four diverse human-centric forgery scenarios: 2,721 audio-driven, 5,611 pose-driven, 4,314 interaction-based, and 5,467 semantic-driven videos (Table 1).
• Gen2Anno’s multi-agent contrastive annotation framework produces omni-annotations integrating binary labels, fine-grained artifact categories, spatial-temporal grounding, and natural-language forgery rationales, providing a richer supervisory signal than prior blind annotation methods.
• Evaluation results show state-of-the-art detectors and large multimodal models struggle with zero-shot generalization and fine-grained anomaly detection on HumanForge (details reported but exact metrics unspecified in abstract).
• Gen2Anno employs six specialized agents coordinating generation provenance analysis and video inspection to avoid false positives from intended prompt-driven edits, significantly improving annotation reliability and explainability.
• The HumanForge benchmark leverages over ten advanced diffusion-based and transformer video generators (Wan2.1, CogVideoX, LTX, OmniWeaving, etc.), reflecting current generative AI capabilities more comprehensively than GAN-centric prior datasets.
• Real source videos from six datasets (HDTF, FFIW, FF++, DFD, SHHQ, TikTok) provide diverse backgrounds, motions, and modalities supporting multi-paradigm generation and evaluation.
• HumanForge preserves heterogeneity by synthesizing videos with variable resolutions and frame rates to better simulate real-world forgery diversity.
• Gen2Anno’s design includes a closed-loop self-correction mechanism that issues review requests for ambiguous annotations, increasing annotation accuracy and confidence.

Threat model

The adversary is an advanced generative model user capable of producing highly realistic human-centric videos, including lip-sync, full-body motion transfer, physical interactions, and semantic edits. They control generation inputs such as source assets, driving conditions, and text prompts. However, the adversary does not have control over or cannot falsify generation provenance metadata accessible to Gen2Anno, which the detection framework leverages to distinguish intentional edits from forgery artifacts.

Methodology — deep read

The authors begin by defining the threat model around adversaries employed to produce realistic human-centric video forgeries leveraging state-of-the-art video diffusion and editing techniques. The adversary can manipulate human-object and human-human interactions, audio-visual synchrony, and semantics but generation provenance metadata is available to the annotation pipeline. The dataset construction sources about 2,000 real videos from public and in-the-wild datasets such as HDTF (high-definition talking faces), DFD (casual motions), FFIW, FF++ (aligned audio extracted), SHHQ (static full body), and TikTok (social dance videos). These provide diverse backgrounds, poses, and modalities.

Using these source assets and driving signals (audio tracks, skeletal poses, text prompts), the authors synthesize 18,849 deepfake videos across four defined paradigms: (1) audio-driven lip sync and facial dynamics, (2) pose-driven full-body motion transfer, (3) complex interaction modeling involving physical contacts and occlusions, and (4) semantic-driven text-to-video editing. These videos are generated by over ten modern generative frameworks primarily based on diffusion models and transformers, including Wan2.1, CogVideoX, LTX-Video, InfiniteTalk, SkyReels, HuMo, OmniWeaving, among others. Outputs maintain variable resolutions (~1280x720), durations (~5 secs), and frame rates to better reflect real-world heterogeneity.

To annotate these videos without labor-intensive manual effort or hallucinated labels, the authors propose Gen2Anno, a multi-agent annotation architecture implemented atop LangGraph. Gen2Anno models annotation as a contrastive verification between an Expected State (E), derived solely from source assets, generation provenance, and driving conditions, and the Actual State (A), derived solely from direct visual/audio inspection of the forged video. The agents operate over a global shared structured state (Gen2AnnoState), keeping input metadata, provenance logs, and intermediate outputs.

The pipeline consists of six specialized agents: 1) Source Profiler extracts invariant source features; 2) Forgery Executor orchestrates generation and records provenance; 3) Director routes scenario-specific expert agents and defines contract constraints on what should remain or change; 4) Reference Analyst (MoE) builds Expected State across audio-visual, intent, kinematic, and semantic sub-experts without access to the forged video; 5) Forgery Inspector (MoE) analyzes the Actual State video using experts on audio-visual sync, physics, kinematics, and spatio-temporal consistency; 6) Chief Judge fuses Expected and Actual States to generate final omni-annotations, accepting only evidence-supported discrepancies for forgery rationales.

Gen2Anno also incorporates a closed-loop self-correction mechanism. If the Chief Judge finds ambiguous evidence, it issues structured review requests to re-invoke relevant agents to refine observations. This iterative process continues until ambiguity resolves or a maximum round limit is reached.

The final outputs are omni-annotations stored in JSON, encoding binary authenticity, artifact categories, spatio-temporal localization, severity, confidence, and natural language logic chains tracing the forgery rationale. Algorithm 1 and Figure 2 illustrate the detailed generation, analysis, verification, and annotation pipeline.

For evaluation, the authors benchmark state-of-the-art traditional deepfake detectors and Large Multimodal Models (LMMs) on HumanForge, observing significant challenges in zero-shot generalization and detailed anomaly reasoning. However, specific metric results or cross-validation details are not explicitly reported in the source. The authors plan public release of all code and dataset upon publication to support reproducibility.

This methodology enables a first-of-its-kind large-scale human-centric multi-scenario deepfake dataset with rich provenance-aware, multi-agent forged video reasoning and annotation, advancing explainable forensics beyond existing static or isolated video datasets.

Technical innovations

• HumanForge is the first large-scale video deepfake benchmark that systematically covers four human-centric forgery paradigms, including complex human-human and human-object physical interactions, overlooked by prior datasets.
• Gen2Anno introduces a novel multi-agent modular framework that contrasts Expected States (derived from generation provenance and references) against Actual States (direct forged video observation), enabling context-aware, precise forgery annotations that avoid false positives from intended edits.
• Integration of a state-centric shared global context (Gen2AnnoState) among six specialized cooperative agents (source profiler, forgery executor, director, reference analyst, forgery inspector, chief judge) for coordinated, scenario-specific fine-grained annotation.
• Incorporation of a closed-loop self-correction mechanism that issues structured review requests to iteratively refine ambiguous annotation decisions, increasing reliability of generated omni-annotations.
• Formalization of forgery rationales as structured tuples containing artifact category, expected state, actual observation, and logical deduction, combining binary classification with spatial-temporal grounding and natural language contrastive reasoning.

Datasets

• HumanForge — 18,849 synthetic videos + ~2,000 real source videos — collected from HDTF, FFIW, FF++, DFD, SHHQ, TikTok datasets

Baselines vs proposed

• State-of-the-art traditional deepfake detectors and large multimodal models (LMMs): zero-shot generalization and fine-grained reasoning significantly challenged by HumanForge dataset (specific metric comparisons not detailed)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08705.

Fig 1: Diagrammatic overview of the proposed HumanForge dataset.

Fig 2: Diagrammatic overview of the proposed Gen2Anno.

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• The evaluation results mention significant challenges but do not provide detailed quantitative metrics, limiting direct comparability or benchmarking reproducibility from the available text.
• While the multi-agent Gen2Anno framework leverages generation provenance, it requires access to detailed generation metadata that may not be available or standardized for all deepfake content in the wild.
• HumanForge videos are synthesized at 1280x720 resolution and standardized durations (~5 seconds), which may not represent all real-world deployment settings with longer or higher resolution video forgeries.
• The complexity of the multi-agent annotation system may limit scalability or adaptability to new generative models without re-tuning the experts or routing strategies.
• No explicit mention of adversarial robustness or evaluation against adaptive forgers attempting to evade Gen2Anno’s contrastive detection.
• The dataset and methods rely primarily on current diffusion and transformer-based models; future generative advances might produce forgeries with fewer detectable inconsistencies, potentially reducing effectiveness.

Open questions / follow-ons

• How well does Gen2Anno generalize to unseen generative models or new forgery paradigms that differ significantly from those used in HumanForge?
• Can the closed-loop multi-agent annotation pipeline be adapted or extended to handle real-world deepfake videos without access to generation provenance metadata?
• How effective is HumanForge and Gen2Anno in detecting adversarially crafted forgeries specifically designed to minimize detectable spatial-temporal or physical inconsistencies?
• Could the multi-agent framework integrate active learning to improve detection models continuously based on new forgery rationales discovered in the wild?

Why it matters for bot defense

HumanForge and the Gen2Anno annotation framework present a significant advance in the robustness and explainability of human-centric deepfake video detection. Bot-defense and CAPTCHA practitioners confronting realistic video-based bot or fraud attacks could leverage the dataset to train and benchmark detectors capable of reasoning about complex human interactions, physical constraints, and semantic consistency beyond face-swapping. The multi-agent, provenance-aware annotation approach offers a blueprint for generating rich forensic explanations that can aid in interpretability and trustworthiness of automated forgery detection systems deployed in security-critical applications. However, practical adoption will require careful consideration of provenance data availability and the ability to handle evolving generative techniques.

Cite

@article{arxiv2607_08705,
  title={ HumanForge: A Human-Centric Deepfake Video Benchmark with Multi-Agent Forgery Rationales },
  author={ Wenbo Xu and Zhimin Chen and Xiaojie Liang and Hengrui Liu and Wei Lu },
  journal={arXiv preprint arXiv:2607.08705},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08705}
}

Read the full paper

• arXiv:2607.08705 (abstract)
• arXiv:2607.08705 (PDF)