FARI: Robust One-Step Inversion for Watermarking in Diffusion Models

Source: arXiv:2607.26723 · Published 2026-07-29 · By Jindong Yang, Han Fang, Weiming Zhang, Nenghai Yu, Kejiang Chen

TL;DR

This paper addresses the critical bottleneck in inversion-based watermarking for diffusion models, where reconstructing the initial noise latent from a generated image is slow and error-prone. Existing methods require many denoising steps (high number of function evaluations, NFE) to reduce truncation errors, but these techniques over-optimize internal inversion accuracy while ignoring external distortions like JPEG compression or cropping, which dominate errors in watermark extraction. The key insight is that the inversion trajectory has markedly lower curvature than the forward generation trajectory, making it highly compressible and amenable to low-NFE (even one-step) approximation. This enables a significant speedup and, more importantly, allows adversarial fine-tuning of the denoiser with LoRA parameters to directly optimize robustness against distortions. Their proposed method, FARI (Fast Asymmetric Robust Inversion), uses one-step distilled inversion plus lightweight adversarial LoRA tuning to extract watermarks robustly and efficiently. Experiments on Stable Diffusion v1.5 and v2.1 models show FARI outperforms 50-step DDIM inversion and other state-of-the-art inversion/acceleration methods in watermark bit accuracy and true positive rate under diverse realistic image distortions while reducing inference time drastically.

Key findings

• The inversion trajectory exhibits substantially lower curvature than the generation trajectory, enabling effective one-step inversion approximation (Fig 1 middle, right).
• FARI surpasses 50-step DDIM inversion robustness under various distortions like JPEG, cropping, resizing, Gaussian noise, blur, and brightness changes on both Stable Diffusion v1.5 and v2.1.
• On Gaussian Shading watermark extraction, FARI achieves bit accuracies above 0.98 on distorted images versus 0.88-0.97 for baselines at much higher NFE (Table 1).
• On Tree-Ring watermarking, FARI obtains TPR close to 1.0 at 10^-3 FPR under strong distortions while 50-step DDIM inversion TPR drops significantly (Table 1).
• FARI reduces inference NFE from 50 to 1, cutting runtime dramatically with roughly 20 minutes of LoRA fine-tuning on a single RTX A6000 GPU.
• Adversarial LoRA training on one-step distilled inversion improves robustness even to unseen distortions not included in the augmentation set (Fig 5 left).
• Increasing LoRA rank yields diminishing returns; rank 8 suffices to balance performance and parameter efficiency (Fig 5 middle).
• Training with more NFE steps (>1) worsens performance slightly and adds latency, confirming one-step inversion is optimal (Fig 5 right).

Threat model

The adversary is a downstream user or external entity attempting to remove, alter, or spoof the embedded watermark in diffusion-generated images by applying common or adversarial image distortions such as JPEG compression, blurring, cropping, noise injection, or resizing. The defender seeks to reliably authenticate images by robust inversion despite these perturbations. The adversary cannot feasibly invert the diffusion process or access the internal model weights with LoRA parameters used in FARI’s inversion branch, nor perform large-scale retraining of the backbone denoiser.

Methodology — deep read

The authors start from the problem that in diffusion model watermarking, inversion of generated images to retrieve embedded watermarks requires reconstructing the initial noise latent. Traditional approaches rely on iterative inversion with many steps (high NFE) to minimize discretization/truncation errors, but in practical scenarios, external distortions dominate the error landscape, making internal precision less critical. Importantly, the inversion trajectory (mapping image back to noise) shows markedly lower curvature compared to the forward generation trajectory, meaning it can be well approximated with fewer steps.

They propose FARI, which performs a one-step inversion distilled from a full DDIM inversion process. Instead of mimicking the 50-step inversion on real images, FARI trains a lightweight LoRA-adapted denoiser branch that maps distorted image latents directly back to the original Gaussian noise in one step. This one-step inversion procedure is much faster and enables end-to-end adversarial fine-tuning on diverse distortion augmentations to improve robustness. The original denoiser weights are frozen, and only the LoRA parameters (low-rank matrices B and A) are updated during training, allowing parameter-efficient updating and avoiding degradation in generation quality.

The training loop samples Gaussian noise vectors and prompts from the MS-COCO dataset, then generates images with a 20-step DDIM schedule. Images are distorted by one of nine augmentation types (JPEG, crop, noise, blur, etc), encoded via VAE encoder into latent space, then inverted with the one-step LoRA branch to reconstruct noise. The loss is the MSE between reconstructed and original noise. The LoRA parameters are trained for 1,000 steps with batch size 4 and learning rate 1e-4 on a single RTX A6000 GPU.

Evaluation compares FARI against many baselines including DDIM (50/1 step), EDICT, BELM, ExactDPM, acceleration solvers like AMED, and generation distillation methods like LCM-LoRA and DMD2. Performance metrics are bit accuracy for Gaussian Shading watermark and true positive rate (TPR) at fixed false positive rate (FPR) 10^-3 for Tree-Ring watermark on 1,000 test prompts from the Stable-Diffusion-Prompts dataset under multiple distortion types.

They further conduct ablations on LoRA rank, training NFE steps, guidance scale, and distortion intensity to study robustness, generalization, and runtime tradeoffs. Reproducibility is supported by released code and pretrained models.

A concrete example end-to-end: a noise vector zT and prompt c are sampled, the 20-step DDIM scheduler generates image X, which is distorted by JPEG compression D. X is encoded to latent zD0, the one-step LoRA inversion model is applied to zD0 producing reconstructed noise estimate ˆzDT. The MSE between zT and ˆzDT is computed and backpropagated to update LoRA parameters ψ. This loop trains FARI to robustly invert distorted images in a single step, enabling high-fidelity watermark extraction with drastic acceleration over multi-step inversions.

Technical innovations

• Identification that DDIM inversion trajectories have significantly lower curvature than generation trajectories, making low-NFE approximation viable for inversion.
• One-step inversion distillation that directly maps images to initial noise without mimicking high-step DDIM inversion outputs, overcoming inherent limitations of iterative inversion.
• Use of lightweight LoRA adapters for adversarial fine-tuning of the denoiser in inversion, preserving original generation quality while enhancing robustness.
• Jointly training one-step inversion and adversarial robustness end-to-end, enabled by drastic NFE reduction and enabling practical adversarial training against diverse distortions.

Datasets

• MS-COCO 2017 — 1,000 prompts for training
• Stable-Diffusion-Prompts (SDP) — 1,000 prompts for evaluation

Baselines vs proposed

• DDIM 50 steps: Gaussian Shading bit accuracy under JPEG distortion = 0.9777 vs FARI 1 step = 0.9834
• DDIM 50 steps: Tree-Ring watermark TPR@1e-3 under random crop = 1.000 vs FARI 1 step = 1.000 (SD v1.5)
• EDICT 50 steps: bit accuracy noisy images ~0.91 vs FARI 1 step ~0.98
• BELM 50 steps: sensitive to guidance scale mismatch; worse TPR and bit accuracy than DDIM baseline
• AMED-Solver (2 steps): competitive bit accuracy (~0.96) but less effective against complex distortions vs FARI
• LCM-LoRA 2 steps: performs worse on inversion robustness than FARI 1 step
• DMD2 1 step: underperforms versus FARI on noisy/distorted images
• ExactDPM (>150 steps): effective for Gaussian noise but computationally prohibitive and fails on distortions causing content loss

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.26723.

Fig 1: Left: Visualization of the inversion error, where latent vectors are projected down to 3

Fig 2: The framework of FARI. FARI simultaneously performs one-step distillation and adver-

Fig 3: Inversion trajectory error of

Fig 4 (page 5).

Fig 5 (page 5).

Fig 6: Visualization of the distortion set used in our experiments. (a) Clean image or identity

Fig 11: FARI reduces reconstruction error, especially under distortion. Naive DDIM single-step

Fig 8 (page 22).

Limitations

• FARI slightly increases internal inversion error on clean distortion-free images compared to 50-step inversion, though this does not harm downstream watermark performance.
• The method depends on LoRA fine-tuning, which requires access to training images and GPUs; not zero-shot.
• Evaluation focuses mainly on common digital distortions; robustness to more complex or adversarial image manipulations is untested.
• Fine-tuning was performed on only 1,000 training prompts, which might limit generalization to diverse generation domains.
• The inversion relies on Stable Diffusion models; applicability to other diffusion architectures or samplers may require adaptation.
• While distortion types during training cover many common corruptions, unforeseen perturbations outside this distribution may degrade performance.

Open questions / follow-ons

• How does FARI perform against sophisticated adversarial attacks explicitly designed to remove or confuse the inversion-based watermark extraction?
• Can the approach be extended to support multi-bit or multi-watermark schemes with similar robustness and efficiency?
• How transferable is the low-curvature inversion property to other diffusion architectures beyond DDIM and Stable Diffusion variants?
• What are the limits of distortion complexity and severity where one-step inversion remains reliable for watermark verification?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, FARI presents a compelling approach to embedding and verifying robust watermarks in diffusion-generated images with practical speed and robustness. The key insight that inversion trajectories can be approximated efficiently with drastically fewer steps without sacrificing robustness could inspire efficient challenge-response schemes where images are uniquely tagged and must be reliably verified even after typical web distortions (compression, resizing, cropping). Moreover, the LoRA-based fine-tuning approach allows incremental robustness improvements without impacting base model operation, facilitating adaptable defenses. However, the method targets watermark extraction and authentication rather than directly mitigating bot manipulation, so it would need integration with detection or challenge issuance layers. Still, its demonstrated resistance against common image corruptions provides a practical foundation for watermark-based image provenance verification in secure content generation and distribution pipelines relevant to anti-fraud or anti-bot systems.

Cite

@article{arxiv2607_26723,
  title={ FARI: Robust One-Step Inversion for Watermarking in Diffusion Models },
  author={ Jindong Yang and Han Fang and Weiming Zhang and Nenghai Yu and Kejiang Chen },
  journal={arXiv preprint arXiv:2607.26723},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.26723}
}

Read the full paper

• arXiv:2607.26723 (abstract)
• arXiv:2607.26723 (PDF)