HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enchancement

Source: arXiv:2607.18217 · Published 2026-07-20 · By Yiyang Cai, Nan Chen, Rongchang Xie, Junwen Pan, Chunyang Jiang, Cheng Chen et al.

TL;DR

The paper addresses the challenge of Human-Object Centric Video Personalization (HOCVP), a subject-driven video generation task that aims to generate videos with multiple humans interacting realistically with objects, including abstract concepts like logos. Current methods struggle with balancing subject fidelity and accurate human-object interactions, especially for inter-subject multi-identity scenarios and intra-subject multi-view or OCR-based references. To tackle these issues, the authors propose HOMIE, a unified HOCVP framework that integrates Multimodal Large Language Model (MLLM) knowledge into video diffusion models differently from prior art. They introduce Global Multimodal Guidance (GMG) that injects MLLM semantic features into video tokens through self-attention, and Modality-Reference Embeddings (MRE) that discriminate input token modalities and bind intra-subject references. Extensive experiments on curated datasets show HOMIE achieves state-of-the-art performance across multiple quantitative metrics including video quality, text adherence, subject consistency, and OCR accuracy. Qualitative and user study results demonstrate its ability to reason abstract references and maintain multi-view consistency.

Key findings

• HOMIE achieves 21.8% relative improvement in OCR accuracy compared to the strongest baseline SkyReels-V3, validating effective OCR map usage.
• On intra-subject multi-view tasks, HOMIE improves DINOrec to 0.696 and DINOacc to 0.560, outperforming baselines like VACE (0.642/0.551) and Phantom (0.633/0.553).
• Ablation shows that removing GMG or MRE lowers performance across text adherence (GMEScore drops from 0.691 to 0.688/0.690) and subject consistency metrics (Face-Sim drops from 0.786 to 0.697/0.738).
• Integrating MLLM features by aligning with the UmT5 text embedding space lowers GMEScore to 0.675 and reduces logo personalization capability.
• User study with 40 participants rates HOMIE as best on overall quality (64.7%), subject consistency (66.1%), and text following (57.9%) compared to 5 strong baselines.
• HOMIE handles multi-human-object interactions and abstract concepts like logos better than prior approaches, successfully attaching logos without explicit prompt mentions.
• GMG enhances self-attention queries and keys of video tokens using pooled global MLLM features to inject temporal multimodal guidance effectively.
• MRE embeddings explicitly tag tokens by modality (video, reference images, MLLM features) and by intra/inter-subject identity to reduce token mixing and improve multi-view consistency.

Threat model

The adversary can supply multiple reference images and textual prompts to condition a video generation model, potentially including abstract or ambiguous objects such as logos. The model must accurately infer latent semantic correspondences across multi-subject and intra-subject references to generate consistent and faithful videos. The adversary does not have access to modify the model internals or training data and cannot bypass the model's multimodal input constraints.

Methodology — deep read

The authors first define the threat model as generating personalized videos from multi-subject and intra-subject references with the challenge of balancing high-fidelity subjects and realistic human-object interactions, including abstract entities like logos. The adversary or task is generating videos conditioned on varied input references (images, OCR maps, textual prompts) with latent semantic correspondences.

Data comes from curated subsets of two open datasets, OpenS2V-5M and PhantomData, filtered for HOCVP relevance. This results in about 300K single-subject samples, 80K multi-subject samples, and an additional curated 20K videos with precise segmentation masks and intra-subject crops. A high-res 40K video subset is also selected. Human and object references come from open source and AIGC-generated sources.

HOMIE's architecture builds on a diffusion transformer (DiT)-based text-to-video model with a 3D Variational Autoencoder (VAE) to encode videos into latents. Text prompts are encoded by a text encoder (UmT5) for cross-modality alignment in the text-to-video backbone. To leverage richer semantic knowledge, HOMIE integrates a Multimodal Large Language Model (MLLM) which takes as inputs the prompt and references, extracting multimodal features from their last hidden states. These are projected by a small MLP into the same token space as the 3D VAE.

A novel multimodal input paradigm concatenates video latents, reference image latents, and aligned MLLM semantic tokens into a single token sequence. HOMIE introduces Global Multimodal Guidance (GMG), which applies a pooling along temporal dimensions on MLLM query/key matrices to create global representations. These global summaries produce affine modulation parameters that rescale and shift the video token queries and keys during self-attention, injecting high-level MLLM semantics into the video token interactions without disturbing reference-image attention patterns.

Additionally, Modality-Reference Embedding (MRE) is introduced. MRE consists of learnable modality embeddings that tag tokens according to their origin—video, image reference, or MLLM features—and separate embeddings that mark tokens by their specific reference entity identity, distinguishing intra-subject (same identity) and inter-subject (different identities). This embedding addition helps disentangle heterogeneous inputs and preserves identity consistency.

Training uses the flow matching objective over noisy latents, conditioning on combined sets of text and multimodal embeddings. A three-stage training regime spans approximately 11K GPU-hours on A100 hardware, sequentially optimizing components from single-subject to multi-subject scenarios.

Evaluation uses a self-curated held-out set of 200 human-object combinations across both inter- and intra-subject personalization settings. Metrics include general video quality (aesthetic and motion smoothness), text-prompt adherence (GMEScore), and subject consistency (face similarity, object similarity via DINO-I and GPT-based measures, plus OCR accuracy). Intra-subject reference evaluations add multi-view consistency metrics DINOrec and DINOacc. Baselines include public and closed-source state-of-the-art T2V personalized video models such as Kling-1.6, VACE, MAGREF, Phantom, and others.

Ablations systematically remove or replace GMG and MRE, and compare different MLLM integration strategies including direct alignment to UmT5 embeddings or full text encoder replacement. Qualitative experiments focus on challenging cases like logo personalization and multi-view consistency. User studies with 40 participants across multiple metrics reinforce quantitative findings.

The authors release training details including architectures, hyperparameters, and datasets (some curated). Exact code or weights are not explicitly stated as released in the source.

Technical innovations

• Global Multimodal Guidance (GMG): Injects pooled global semantic MLLM features into video token self-attention queries and keys through affine modulation, enabling deeper cross-modal interactions without disrupting reference-image attention patterns.
• Modality-Reference Embedding (MRE): Learnable embeddings that simultaneously identify token modality (video, reference images, MLLM features) and distinguish intra- and inter-subject references to reduce identity confusion and enforce consistent multi-view alignment.
• A unified multimodal input paradigm that concatenates video token latents, reference image token latents, and MLLM extracted semantic tokens into a single input sequence, balancing text encoder controllability and rich multimodal knowledge injection.
• A novel MLLM integration strategy preserving the original text encoder while allowing the MLLM to focus exclusively on extracting reference relationships via multimodal input fusion, avoiding costly re-alignment or controller degradation.

Datasets

• OpenS2V-5M — ~5 million samples — open-source subject-driven video generation dataset
• PhantomData — large-scale subject-consistent video generation dataset — open-source
• Self-curated HOCVP dataset — 300K single-subject samples plus 80K multi-subject samples filtered from above datasets
• High-quality curated subset — 20K HOC videos with segmentation masks for humans and objects
• Filtered high-res selection — ~40K 720p samples from multi-subject data

Baselines vs proposed

• SkyReels-V3: OCR accuracy = 0.376 vs HOMIE: 0.452 (21.8% relative improvement)
• VACE: DINOrec = 0.642, DINOacc = 0.551 vs HOMIE (Wan2.2): 0.696, 0.560
• Plain model (no MLLM): GMEScore = 0.682, Face-Sim = 0.741 vs HOMIE: 0.691, 0.786
• MLLM to UmT5 integration: GMEScore = 0.675 vs HOMIE GMG integration: 0.691
• Phantom: OCR accuracy ≈ 0.302 vs HOMIE: 0.452

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.18217.

Fig 1: HOMIE addresses HOCVP with both inter/intra-subject references: (1) multi-human-

Fig 2: Framework design of MLLM-facilitated video personalization.

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The study focuses on curated and filtered datasets; real-world diversity and unseen identities may challenge model generalization.
• No explicit adversarial evaluation against maliciously crafted references or deliberately confusing inputs to test robustness.
• Training requires large computation (11K GPU-hours), potentially limiting reproducibility or deployment in low resource settings.
• Some reliance on AIGC-generated references for human subjects may bias evaluation metrics like face similarity.
• No official release of code or pre-trained weights disclosed, limiting external validation and extension.
• Evaluation is primarily quantitative/perceptual; formal statistical hypothesis testing and cross-validation details are limited.

Open questions / follow-ons

• How robust is HOMIE to adversarial or out-of-distribution reference images designed to confuse modality-reference embeddings?
• Can the model generalize to zero-shot unseen object categories or extremely abstract concepts beyond logos?
• How does HOMIE perform with longer videos or more complex temporal interaction dynamics beyond the datasets used?
• What is the effect of scaling MLLM sizes or adding additional modalities like audio on HOCVP quality?

Why it matters for bot defense

For bot-defense and CAPTCHA engineers, HOMIE’s advances in multimodal conditional synthesis underscore the growing sophistication of AI systems that understand complex semantic references and subtle multimodal cues. In particular, the paper demonstrates that leveraging MLLMs jointly with video diffusion models via carefully designed integration strategies like GMG and MRE enables richer, more precise control over generated content involving humans, objects, and abstract elements. This suggests that future bot challenges involving personalized video manipulations or multimodal context reasoning must account for emerging capabilities like these.

From a CAPTCHA perspective, the methods highlight the importance of robust semantic grounding when enforcing controls against bots generating or interpreting human-object interaction videos. Detecting misuse or spoofing may require analyzing latent cross-modal token relationships and exploiting inconsistencies in multimodal embeddings that HOMIE specifically designs to handle. Thus, engineers should consider both how to use such models practically and how to devise countermeasures or detection mechanisms that scrutinize semantic alignment across modalities in human-object video content.

Cite

@article{arxiv2607_18217,
  title={ HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enchancement },
  author={ Yiyang Cai and Nan Chen and Rongchang Xie and Junwen Pan and Chunyang Jiang and Cheng Chen and Wen Zhou and Zhenbang Sun and Wei Xue and Wenhan Luo and Yike Guo },
  journal={arXiv preprint arXiv:2607.18217},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.18217}
}

Read the full paper

• arXiv:2607.18217 (abstract)
• arXiv:2607.18217 (PDF)