Diff-ID: Identity Consistent Facial Image Generation and Morphing via Diffusion Models

Source: arXiv:2607.25078 · Published 2026-07-27 · By Taimoor Rizwan, Sara Atito, Muhammad Awais, Zhenhua Feng, Josef Kittler

TL;DR

This paper addresses the challenge of preserving identity consistency in high-resolution facial image generation using diffusion models, a critical problem for biometric authentication, forensic analysis, and privacy-sensitive applications. The authors introduce Diff-ID, a diffusion-based framework that integrates multi-modal embeddings—identity embeddings from ArcFace and semantic embeddings from CLIP—through a custom dual cross attention adapter in a fine-tuned Stable Diffusion UNet. To promote identity fidelity during generation, they propose a pseudo discriminator loss weighted exponentially over diffusion timesteps, enforcing stronger identity constraints as noise decreases. Additionally, they develop a unified morphing pipeline leveraging DDIM inversion and embedding interpolation to enable smooth identity morphing between faces without per-identity fine tuning. They curate and caption a large 210K image dataset from CelebA-HQ, FFHQ, and LAION-Face to strengthen identity-aware training.

Empirical results show Diff-ID achieves a superior trade-off between identity preservation and image realism compared to baselines. While the raw ArcFace face similarity is marginally below InstantID, Diff-ID obtains substantially lower Fréchet Inception Distance (FID) scores and the highest Face Image Quality (FIQ) scores, a new combined metric the authors introduce linking identity similarity with perceptual realism. Qualitative morphing results demonstrate smooth transitions between identities through latent space interpolation without expensive fine tuning or model switching. The work highlights the importance of jointly evaluating identity and realism rather than considering them in isolation for robust facial image synthesis.

Key findings

• Diff-ID trained on a 210K image dataset curated from CelebA-HQ, FFHQ, and LAION-Face, improving identity-aware generation coverage.
• ArcFace and CLIP embeddings fused via a dual cross attention adapter in Stable Diffusion UNet preserve fine-grained facial features like jawline and nose shape.
• Pseudo discriminator identity loss weighted exponentially by diffusion timestep enhances identity preservation during late denoising stages.
• Diff-ID achieves lower FID scores compared to InstantID and IP Adapter on held-out face datasets, indicating better perceptual realism.
• Face Similarity (ArcFace cosine) for Diff-ID is competitive but slightly lower than InstantID, underscoring trade-offs.
• Face Image Quality (FIQ), combining identity and realism into a ratio score, is highest for Diff-ID among evaluated methods, reflecting balanced performance.
• The morphing pipeline using DDIM inversion plus spherical interpolation of embeddings enables smooth identity blends without checkpoint swapping.
• Training converged after 1,000,000 steps on 4x NVIDIA A100 GPUs in 72 hours with Adam optimizer and batch size 16.

Threat model

The adversary is anyone exploiting synthetic facial generation to distort or manipulate identity such that biometric systems fail or are deceived. The attacker cannot access internal model parameters or retrain Diff-ID but may attempt to generate identity-ambiguous images to evade recognition. Diff-ID assumes the generation pipeline is unaltered post-training and relies on embedding-based identity constraints rather than external discriminators or adversarial defenses.

Methodology — deep read

The authors first define the threat model as a need for robust identity preservation in synthetic facial images to support biometric security and privacy compliance; adversaries may try to exploit identity drift but cannot access model internals or retrain it. They compile a large, cleaned dataset of 210K aligned 512x512 face images curated from CelebA-HQ, FFHQ, and LAION-Face datasets. Since FFHQ and LAION-Face lacked textual captions, they fine-tune a BLIP model on CelebA-HQ attribute vectors converted to captions, generating descriptive prompts for all images enhancing semantic richness without explicit attribute control.

Diff-ID builds on Stable Diffusion 1.5 UNet architecture working in latent space for efficient high-fidelity image generation. For identity preservation, it extracts 512-d ArcFace embeddings capturing fine geometric identity cues, and CLIP embeddings capturing semantic context from the input image and prompt. The two modalities are separately projected into a common latent space by learnable matrices, concatenated with max and average pooled features to form a final identity representation. This is processed alongside the CLIP text embedding through a dual cross attention mechanism: one branch uses identity queries attending over text keys/values and the other uses text queries attending over identity keys/values, enabling reciprocal enrichment. No softmax normalization is applied to preserve feature magnitudes.

The dual attentions outputs are concatenated and fused through a multilayer perceptron with ReLU nonlinearities, producing a refined embedding that modulates denoising in the UNet. The training loss combines the standard diffusion denoising loss (predicting noise in latent space) and a pseudo discriminator identity loss computed as 1 minus the cosine similarity between ArcFace embeddings extracted from original and generated images. The identity loss is dynamically weighted by an exponential decay function of the diffusion timestep, emphasizing identity preservation primarily in later, low noise steps. The overall loss balances denoising and identity with λ_identity=0.1.

For morphing, the authors exploit deterministic DDIM inversion to map two source images into noise latent space, and spherical linear interpolate their fused ArcFace+CLIP embeddings plus linearly interpolate their noise latents. These blended latents and embeddings serve as conditions for DDIM deterministic forward sampling through the Diff-ID model, generating smooth identity interpolations without checkpoint swapping or additional fine tuning.

Training used Adam with learning rate 1e-5, batch size 16, mixed precision, and gradient checkpointing across 4 NVIDIA A100 GPUs for ~72 hours (1M steps). The VAE encoder/decoder and ArcFace/CLIP models were frozen to retain pretrained representations. Evaluation uses the held-out face datasets with both qualitative side-by-side comparisons and quantitative metrics: ArcFace cosine similarity (Face Similarity, FS) measures identity retention, Fréchet Inception Distance (FID) assesses perceptual fidelity, and a novel Face Image Quality (FIQ) metric combines FS/FID as a ratio to explicitly quantify the identity-realism trade-off.

Technical innovations

• Integration of fine-grained ArcFace identity embeddings with CLIP semantic embeddings through a dual cross attention adapter within Stable Diffusion UNet.
• Exponential dynamic weighting of ArcFace cosine similarity identity loss aligned with diffusion timesteps to emphasize identity preservation during denoising.
• A curated, captioned 210K image dataset combining CelebA-HQ, FFHQ, and LAION-Face leveraging fine-tuned BLIP for semantic prompt augmentation.
• Unified DDIM latent inversion and spherical embedding interpolation morphing pipeline enabling smooth, identity consistent face morphs without model fine tuning or checkpoint swapping.
• Introduction of Face Image Quality (FIQ), a complementary metric combining ArcFace similarity and FID to jointly evaluate identity realism trade offs.

Datasets

• CelebA-HQ — ~30K images — public
• FFHQ — ~70K images — public
• LAION-Face — ~110K images — subset of LAION dataset, filtered and captioned
• Combined curated dataset: 210K images (cleaned, resized, captioned) — non-public composite

Baselines vs proposed

• InstantID: Face Similarity (FS) = higher than Diff-ID, FID = higher than Diff-ID
• IP Adapter: FID = higher than Diff-ID, FS = lower than Diff-ID
• Diff-ID: FID = lowest among evaluated models, FS = competitive but slightly lower than InstantID
• Diff-ID: Face Image Quality (FIQ) = highest among all baselines, showing best identity-realism trade-off

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.25078.

Fig 1: Data pipeline used to curate and caption the identity centric dataset.

Fig 3: Illustration of the exponential timestep weighting Wt as a function of the normalized timestep τ. Identity

Fig 4: DiffID Morph: identity preserving morphing pipeline built on DDIM inversion and dual embedding

Fig 5: Face Similarity (FS) for one subject under four stylistic variations. FS remains high even for non-photorealistic

Fig 6: Identity-specific comparison for one subject (ID1) across different models. Metrics (FS, FID, and FIQ) are

Fig 8: illustrates three morphing strategies: (i) embedding only interpolation of fused identity vectors, (ii) DiffID

Fig 7 (page 11).

Fig 8 (page 11).

Limitations

• Diff-ID does not surpass InstantID on raw ArcFace similarity, indicating a slight compromise on pure identity fidelity for better realism.
• No explicit or fine-grained attribute editing mechanism is provided; the model does not support precise control over pose or expression changes.
• The morphing evaluation is primarily qualitative; quantitative assessment of morph realism or biometric vulnerability is not comprehensively reported.
• The curated dataset and trained model weights are not publicly released, limiting exact reproducibility.
• No adversarial robustness or attack evaluation to test identity preservation under adversarial perturbations.
• The exponential weighting hyperparameter k and loss-balancing λ_identity are fixed; sensitivity analysis or ablation on these choices is limited.

Open questions / follow-ons

• How does Diff-ID perform under adversarial attacks targeting identity features during diffusion sampling?
• Can the exponential dynamic weighting scheme be optimized or adapted online to improve identity preservation?
• What are quantitative biometric impacts of morphing trajectories generated by Diff-ID on real-world recognition systems?
• How would the model behave when extended for explicit attribute or expression editing alongside identity preservation?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, controlling identity fidelity in generated faces is crucial when synthetic images are used for training biometric anti-spoofing systems or for forensic validation. Diff-ID's approach of fusing ArcFace and CLIP embeddings to balance photorealism with identity consistency offers a pathway to create higher fidelity synthetic datasets that can better mimic real biometric variation without sacrificing identity—key for robust biometric CAPTCHA systems resisting morphing or spoofing attacks. Moreover, the morphing pipeline enables generating synthetic face blends useful for testing biometric model vulnerabilities to morph attacks or adversarial examples. The Face Image Quality metric also provides a more holistic evaluation tool combining identity preservation with perceptual realism, which is important when assessing synthetic data quality in security contexts. However, its lack of public weights and limited adversarial evaluation mean direct adoption will require further adaptation and rigorous testing.

Cite

@article{arxiv2607_25078,
  title={ Diff-ID: Identity Consistent Facial Image Generation and Morphing via Diffusion Models },
  author={ Taimoor Rizwan and Sara Atito and Muhammad Awais and Zhenhua Feng and Josef Kittler },
  journal={arXiv preprint arXiv:2607.25078},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.25078}
}

Read the full paper

• arXiv:2607.25078 (abstract)
• arXiv:2607.25078 (PDF)