AnyBokeh: Physics-Guided Any-to-Any Bokeh Editing with Optical Fingerprint Transfer

Source: arXiv:2606.31959 · Published 2026-06-30 · By Xinyu Hou, Xiaoming Li, Zongsheng Yue, Chen Change Loy

TL;DR

AnyBokeh addresses the challenging problem of post-capture depth-of-field (DoF) manipulation from a single image taken under arbitrary focus and aperture settings. Unlike prior methods that assume an all-in-focus (AIF) input or reconstruct such an image as an intermediate step, AnyBokeh directly estimates the source blur state using a signed circle-of-confusion (CoC) map and disparity map. It then models the linear relationship between CoC and disparity difference to extract a source-specific optical fingerprint, enabling physics-guided transfer of optical characteristics to any desired focus and aperture settings. A generative editor conditioned on both source and target CoC maps performs relative blur synthesis, allowing spatially adaptive deblurring, preservation, and defocus rendering. This avoids the 'AIF bottleneck' that discards valuable blur cues and eliminates test-time blur-level calibration common in existing works. The authors further create a large-scale, high-fidelity synthetic dataset (UnrealBokeh) with accurate depth, camera settings, and CoC ground truth to physically supervise training. Experiments on real-world benchmarks demonstrate that AnyBokeh faithfully controls DoF edits under any-to-any, all-in-focus-to-bokeh, and defocus-to-AIF settings, while outperforming prior deblur-then-render pipelines by preserving optical consistency and producing more visually pleasing, artifact-free results.

Key findings

• AnyBokeh’s optical fingerprint estimation transfers source CoC scale to target settings without requiring full camera metadata or test-time blur calibration.
• Stage 1 CoC and disparity estimation jointly trained on UnrealBokeh achieves consistent blur and geometry maps across aperture changes (Fig. 3).
• Dual-CoC conditioning in Stage 2 enables spatially adaptive relative blur synthesis, improving over baselines that deblur then reblur separately (Fig. 4).
• AnyBokeh handles inputs with existing defocus, unlike prior methods restricted to all-in-focus inputs or requiring intermediate AIF reconstruction.
• UnrealBokeh contains 11,477 rendered images and 382,888 source-target editing pairs with complete depth and optical metadata.
• Training with signed logarithmic CoC normalization yields better accuracy for near-focus regions critical for realistic refocusing control.
• On benchmarks like EBB! and RealBokeh, AnyBokeh improves perceptual metrics (LPIPS, DISTS) and realism (FID) versus DRBNet + BokehMe and Restormer + BokehDiff baselines.

Threat model

n/a — The paper targets aesthetic image editing and physically guided blur synthesis rather than security or adversarial threat modeling. Adversaries are not explicitly modeled; instead, the framework assumes access to the source defocused image with unknown camera settings and aims for faithful bokeh edits.

Methodology — deep read

The AnyBokeh pipeline consists of two core stages guided by physically grounded blur modeling.

1. Threat Model & Assumptions: The framework targets post-capture editing of images with unknown and arbitrary source focus and aperture settings. The adversary is not explicitly considered here; the problem is posed as accurately modeling and transferring physical lens blur for aesthetic image editing. Camera parameters like aperture and focal length may be unavailable, but the input image is assumed to contain blur cues correlated with scene geometry.

2. Data: To train the system, UnrealBokeh, a large synthetic dataset, is constructed using Unreal Engine rendering diverse indoor/outdoor scenes. It comprises 11,477 RGB images with dense ground-truth depth, complete camera EXIF data (focus distance, aperture, focal length, sensor size), and corresponding signed CoC maps computed from the thin-lens model. The dataset forms 382,888 source-target editing pairs by varying focus and aperture while holding geometry fixed. A train/validation split on environment-camera groups prevents leakage.

3. Architecture & Algorithm:

   • Stage 1 jointly predicts a signed source CoC map (representing spatially varying blur diameter with sign indicating in front/behind focal plane) and disparity map from the input defocused image. The estimation uses a generative inpainting formulation with a strong image prior (FLUX.1-Fill-dev1 backbone fine-tuned with LoRA). The input is arranged as a horizontal triptych (source image plus CoC and disparity maps as masked regions to inpaint).
   • The predicted CoC and disparity maps are resized to original resolution, with CoC scaled appropriately. CoC values are normalized with signed logarithmic transformation to prioritize accuracy near focus boundaries.
   • An optical fingerprint κ_src is estimated as the slope of the linear relationship CoC_px = κ * (D_focus - D), relating the predicted CoC and disparity maps. This fingerprint captures the source optical scale derived from camera and imaging parameters, but learned directly from the input blur state.
   • Target CoC maps for desired focus and aperture are analytically computed by scaling κ_src by the aperture ratio and adjusting the focal disparity, without requiring full metadata or test-time calibration.

4. Stage 2 uses a generative editor conditioned on source image, source CoC, and target CoC (dual-CoC conditioning) to synthesize the final target bokeh image. CoCs are normalized with signed tanh for numerical stability. The editor operates on overlapping patches with Hann-windowed merging to preserve resolution and avoid tiling artifacts.

5. Training regime: Both stages are fine-tuned for approximately 30K iterations on 4 NVIDIA A100 GPUs with batch size 1, learning rate 1e-4, AdamW optimizer, and offset noise. Stage 1 is trained as an inpainting regression task supervised with physically accurate CoC and disparity ground truth. Stage 2 is trained on aligned source-target pairs to learn relative blur synthesis.

6. Evaluation: Tested on real datasets EBB! and RealBokeh, using LPIPS, DISTS perceptual metrics, and FID for realism, under three settings: any-to-any bokeh editing, AIF-to-bokeh rendering, and defocus deblurring (bokeh-to-AIF). Baselines include cascaded deblur-then-render (Genfocus), DRBNet, and Restormer architectures combined with state-of-the-art renderers. Qualitative examples demonstrate better boundary handling, preservation, and artifact reduction. Ablations confirm dual-CoC conditioning importance and normalization strategies.

7. Reproducibility: Code and dataset are publicly released at https://github.com/itsmag11/AnyBokeh. UnrealBokeh dataset includes full optical parameters for supervised learning. Specific seeds or hyperparameter search details are not mentioned. The synthetic dataset allows full traceability of physically grounded supervision.

One concrete example: An input image with unknown aperture and focus containing spatially varying blur is fed into Stage 1, predicting crisp CoC and disparity maps. The optical fingerprint κ_src slope is fitted from these maps. Given a desired aperture change and focus plane, the target CoC map is computed by scaling κ_src and adjusting focal disparity. The generative editor then synthesizes the exact per-pixel relative blur adjustment to produce a realistic bokeh image, preserving sharp regions and defocusing others, avoiding artifacts commonly introduced by forced deblurring.

Technical innovations

• Formulating any-to-any bokeh editing via explicit estimation and transfer of a source-specific optical fingerprint linking circle-of-confusion and disparity rather than reconstructing an all-in-focus intermediate.
• Introducing dual-CoC conditioning that inputs both source and target blur states to the generative editor, enabling spatially adaptive relative blur synthesis for preservation, sharpening, and defocus without hallucination.
• Creating UnrealBokeh, a physically accurate synthetic dataset with complete depth, camera parameters, and ground-truth signed CoC maps, enabling supervised training of CoC and disparity estimation.
• Using a signed logarithmic normalization for CoC regression to emphasize near-focus blur accuracy critical for realistic refocusing boundaries.

Datasets

• UnrealBokeh — 11,477 images, 382,888 editing pairs — synthetic Unreal Engine renderings with ground-truth depth, complete camera EXIF metadata, and computed CoC maps
• EBB! — 9,774 images — real dataset, partial focus and aperture variation, no CoC ground truth
• RealBokeh — 27,451 images — real-world dataset with paired focus/aperture variations but no ground-truth depth or CoC

Baselines vs proposed

• DRBNet + BokehMe: LPIPS = 0.32 vs AnyBokeh: 0.26
• Restormer + BokehDiff: DISTS = 0.19 vs AnyBokeh: 0.16
• Genfocus (DeblurNet + BokehNet): introduces artifacts and hallucinations, quantitatively outperformed by AnyBokeh in LPIPS and FID
• Ablation without dual-CoC conditioning leads to perceptually blurrier outputs and inaccurate refocusing

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31959.

Fig 1: Overview of the AnyBokeh framework. (1) Stage 1 predicts the signed source CoC map

Fig 2: Samples from UnrealBokeh. For the same (environment, camera, focal length) scene,

Fig 3 (page 5).

Fig 4 (page 5).

Fig 5 (page 5).

Fig 3: Qualitative Stage 1 predictions on EBB! and RealBokeh. Each row shows the same scene

Fig 7 (page 6).

Fig 8 (page 6).

Limitations

• Evaluation focuses on perceptual / reference-based metrics; robustness to extreme out-of-distribution optical and scene conditions is not fully tested.
• The synthetic UnrealBokeh dataset, while high-fidelity, may not capture all real optical aberrations and sensor noise characteristics in wild imagery.
• Source CoC and disparity estimation can be ambiguous near complex depth discontinuities or severe blur where visual cues are weak.
• Absolute CoC control mode is provided, but accurate user specification remains challenging without metadata.
• Performance depends heavily on the accuracy of disparity and CoC predictions; failure cases under unknown domain shifts are not exhaustively reported.
• Training and inference rely on multi-GPU setups and are computationally intensive, potentially limiting real-time applications.

Open questions / follow-ons

• How robust is the optical fingerprint estimation in the presence of complex optical aberrations, sensor noise, or novel lens designs not represented in training data?
• Can the framework be extended to handle video sequences maintaining temporal coherence in blur editing under changing focus and aperture?
• To what extent can the approach be adapted to work without any disparity supervision, e.g., using self-supervised or monocular depth estimation in wild images?
• How would the method perform when applied to real images with motion blur or other mixed degradations?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, AnyBokeh provides insights into accurate modeling and manipulation of spatially varying defocus blur tied to physical optics and scene geometry. Such fine-grained blur synthesis and removal, grounded in interpretable physical parameters, may inform robustness evaluation or synthetic challenge creation in visual CAPTCHA schemes. The dual-CoC conditioning and optical fingerprint transfer introduce powerful methods for generating or detecting realistic depth-of-field effects that could be used either to spoof human-like photographic artifacts or to build blur-based defenses against automated recognition. Additionally, the physics-guided blur modeling offers a framework potentially useful for authenticating genuine photographic content versus generated or manipulated imagery, a growing concern for bot-detection in media platforms. However, the reliance on accurate depth and blur estimation highlights the challenges for fully automated bokeh editing in unconstrained real-world imaging conditions, which may limit direct applicability but opens interesting directions for adversarial or robustness research in visual challenge design.

Cite

@article{arxiv2606_31959,
  title={ AnyBokeh: Physics-Guided Any-to-Any Bokeh Editing with Optical Fingerprint Transfer },
  author={ Xinyu Hou and Xiaoming Li and Zongsheng Yue and Chen Change Loy },
  journal={arXiv preprint arXiv:2606.31959},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.31959}
}

Read the full paper

• arXiv:2606.31959 (abstract)
• arXiv:2606.31959 (PDF)