BG-REAL: A Public Real-Data Anchored Benchmark for Background Manipulation Detection and Localization

Source: arXiv:2607.26232 · Published 2026-07-28 · By Bugra Alperen Uluirmak, Rifat Kurban

TL;DR

BG-REAL introduces a public benchmark specialized for evaluating background manipulation detection and localization in images, addressing a gap in existing forensics datasets focused mostly on foreground-centric edits like copy-move or splicing. The benchmark is grounded on real data from Open Images V7, supplemented with synthetic controls, totaling 7,000 samples across 1,200 source groups. It categorizes manipulations into six edit families, including authentic, matched authentic controls, classic and harmonized composites, public background replacement, and JPEG/resize robustness variants. Unlike prior datasets, BG-REAL emphasizes matched authentic controls to isolate re-encoding artifacts from true manipulation evidence, source-disjoint splits to prevent train-test leakage, and multi-axis evaluation conditions probing generalization and shortcut vulnerabilities. Benchmark results from three external state-of-the-art baselines (TruFor, MVSS-Net, HiFi-Net) and an internal CNN model (BG-RIFT) reveal strong domain gaps and widespread false-positive errors triggered by re-encoding artifacts rather than genuine manipulations. The benchmark provides a reproducible pipeline, thorough quality audits, and varied metrics for image-level detection and pixel-level localization.

Key findings

• The BG-REAL benchmark contains 7,000 samples from 1,200 source groups, including 6,000 real Open Images V7 anchored samples and 1,000 synthetic control samples.
• Matched-authentic controls isolate re-encoding effects and show false-positive rates at the fixed validation threshold ranging from 0.57 (TruFor) to 1.00 (HiFi-Net, RGB, Artifact baselines), indicating systemic shortcut risks.
• Image-level detection AUROCs on held-out in-distribution splits: TruFor 0.838 ± 0.029, BG-RIFT 0.713 ± 0.043, MVSS-Net 0.566 ± 0.051, HiFi-Net 0.538 ± 0.018, showing a large domain gap for external baselines without fine-tuning.
• Localization (pixel F1) performance varies by baseline and split, with TruFor consistently producing stronger masks (e.g. 0.414 pixel F1 on ID) compared to BG-RIFT (0.343) and MVSS-Net (0.329).
• BG-RIFT’s fixed high-pass residual branch improves image-level AUROC by ~5 percentage points and pixel F1 by ~10 points versus an RGB-only ablation (e.g., test_id AUROC 0.767 vs. 0.713).
• Source-disjoint splits and multiple out-of-distribution views (background, generator, tool, robustness) provide nuanced evaluation axes; tool-OOD is a zero-training-leakage synthetic-only check covered only by BG-RIFT.
• Human-assisted quality control reviewed 599 samples using an AI-assisted single-reviewer protocol, flagging 496 mask issues but retaining them as audit signals, not exclusions.
• Statistical tests across five seeds show significant AUROC differences favoring TruFor over BG-RIFT and BG-RIFT over MVSS-Net (Holm-corrected p < 0.05).

Threat model

The adversary is a practical image forger who performs subtle background manipulations—such as background replacement or harmonization—to deceive visual inspection and forensic detectors. They can produce visually plausible, high-quality edits that blend lighting, color, and perspective cues, making traditional boundary or foreground-centric detectors ineffective. The adversary’s manipulations may be combined with common post-processing steps like JPEG compression and re-encoding. They are not modeled as directly adaptive or white-box attackers targeting specific detection models, but as creators of broadly challenging forgery content.

Methodology — deep read

1. Threat model & assumptions: The paper assumes a forensics evaluation setting focusing on real-world background manipulation adversaries who produce visually plausible edits difficult to detect by boundary or foreground-centric cues. The main adversarial challenge is detecting subtle background edits that blend lighting, texture, and color, especially under post-processing like re-encoding. The adversary’s specific capabilities are not formally defined; the focus is on the benchmark’s ability to capture weaknesses in detection methods against such manipulations rather than adversarial attack-resistance per se.

2. Data: BG-REAL sources images from Open Images V7 with instance segmentation masks. Starting from 1,000 unique raw public-source images, these expand into 1,200 processed source groups containing 7,000 samples: 6,000 are anchored in Open Images data and 1,000 are synthetic controls created via manipulation pipelines. Six edit families are defined: authentic (unmodified), matched authentic control (re-encoded authentic), classic composite, harmonized composite, public background replacement, and JPEG/resize robustness variants. A source-disjoint train/validation/test split over groups is enforced to prevent leakage. Matched authentic controls share processing pipelines to isolate artifacts due to encoding versus genuine manipulation evidence.

3. Architecture/algorithm: The core internal model, BG-RIFT, is a compact convolutional neural network with two branches: an RGB branch and a fixed high-pass residual branch. These features are fused using a soft region proposal head segmenting into foreground/background/boundary regions. BG-RIFT produces three outputs per image: a pixel-level mask of manipulated areas, an image-level manipulation authenticity label, and a six-way edit-family classification. The model is trained with Adam optimizer, using a learning rate of 10⁻³, batch size effectively 16 (including gradient accumulation), and input image resolution 256×256. A matched-authentic contrastive loss component was implemented but not enabled in training at this release time.

4. Training regime: BG-RIFT is trained for 8 epochs over the BG-REAL training partition with hyperparameters as above. Five random seeds are used for evaluation to assess variability. RGB-only ablation experiments remove the residual branch to test its utility with a fixed seed (42).

5. Evaluation protocol: The evaluation measures image-level detection using threshold-free AUROC and AUPRC, plus threshold-dependent F1 and accuracy fixed once on the validation split to avoid optimistic bias. Localization metrics include pixel-level IoU, pixel F1, mask average precision, and boundary F1 at fixed edge tolerances. Multiple evaluation splits probe in-distribution held-out groups, source-out-of-distribution groups, background- and generator-OOD, tool-OOD synthetic samples, matched authentic controls, and JPEG/resize robustness variants. Baselines evaluated zero-shot without fine-tuning include external TruFor, MVSS-Net, and HiFi-Net models with official released weights and adaptations into BG-REAL’s prediction format. Statistical significance uses paired t-tests corrected with Holm-Bonferroni, with bootstrap 95% confidence intervals reported over five seeds.

6. Reproducibility: The release includes publicly available code for dataset construction, evaluation pipeline, model adapters, and evaluation seed controls, plus detailed quality control logs and human-assisted annotations. The original Open Images data are referenced but not redistributed. Baseline weights come from official repositories but are used zero-shot without BG-REAL-specific tuning. The paper provides a source and sample manifest to audit data provenance and grouping.

Technical innovations

• A novel matched-authentic control mechanism that pairs authentic but re-encoded images to distinguish re-encoding artifacts from true manipulation evidence in evaluation.
• A focused six-family taxonomy for background manipulation, differentiating classic and harmonized composites, public background replacement, and robustness post-processing variants.
• A large-scale, publicly anchored dataset construction pipeline producing 7,000 samples with source-disjoint train/val/test splits and extensive mask- and leakage-quality audits.
• An evaluation protocol probing shortcut vulnerabilities via matched control false-positive measurements and multi-axis out-of-distribution conditions, including tool-OOD synthetic-only zero-leakage checks.

Datasets

• BG-REAL — 7,000 samples over 1,200 source groups — built on Open Images V7 instance segmentation with 6,000 real anchored and 1,000 synthetic control samples

Baselines vs proposed

• TruFor: test_id AUROC = 0.838 ± 0.029 vs BG-RIFT test_id AUROC = 0.713 ± 0.043
• TruFor: source_OOD AUROC = 0.842 ± 0.029 vs BG-RIFT source_OOD AUROC = 0.729 ± 0.032
• MVSS-Net: test_id AUROC = 0.566 ± 0.051 vs BG-RIFT test_id AUROC = 0.713 ± 0.043
• HiFi-Net: test_id AUROC = 0.538 ± 0.018 vs BG-RIFT test_id AUROC = 0.713 ± 0.043
• BG-RIFT ablation (RGB-only): test_id AUROC = 0.713 vs BG-RIFT full model test_id AUROC = 0.767 (single seed)
• Matched-authentic-control false-positive rates: TruFor 0.57, BG-RIFT 0.64, MVSS-Net 0.99, HiFi-Net 1.00

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.26232.

Fig 1: BG-REAL construction and evaluation pipeline.

Fig 2: Example BG-REAL edit family panels for two distinct public source groups (a food

Fig 3: Dataset composition by edit family and source family. The public background re-

Fig 4: Human-assisted quality-control score distributions for the reviewed subset.

Fig 5: Split allocation and QA audit summary.

Fig 6: Five-seed baseline performance on ID and source-OOD splits. The AUROC panels

Fig 7: Localization quality (mean pixel F1, mean IoU), pooled across TruFor, MVSS-Net,

Fig 8: Left: false-positive rate (1−accuracy) on the matched-authentic-control split, where

Limitations

• External baselines (TruFor, MVSS-Net, HiFi-Net) were evaluated zero-shot without BG-REAL-specific fine-tuning, likely limiting their domain adaptation and performance.
• Human-assisted quality control relies on a single reviewer with AI pre-scoring, resulting in no inter-rater reliability measures and potential reviewer anchoring bias.
• Some evaluation splits like background-OOD and generator-OOD are tag-filtered subsets of the same held-out test partition rather than independently sourced populations, limiting claims about generalization.
• Matched-authentic control analyses reveal that re-encoding artifacts cause high false-positive rates even in best baselines, suggesting a common shortcut not yet fully mitigated.
• The matched-authentic contrastive loss in BG-RIFT was implemented but not yet used during training; its potential impact remains unexplored.
• The current release does not include diffusion-generated background manipulations, planned as future work, limiting evaluation coverage for emerging forgery methods.

Open questions / follow-ons

• How to design models intrinsically robust to re-encoding artifacts that currently cause widespread false positives in matched authentic controls?
• What is the impact of fine-tuning the external baselines (TruFor, MVSS-Net, HiFi-Net) on BG-REAL, and can domain adaptation close the observed performance gaps?
• How will diffusion-based background manipulations, planned for future BG-REAL extensions, affect detectability and model evaluation?
• Can matched-authentic contrastive training losses improve BG-RIFT or other models’ ability to disentangle manipulation evidence from processing artifacts?

Why it matters for bot defense

For bot-defense engineers working with CAPTCHA and related anti-automation challenges, BG-REAL provides insights into the detection of subtle, plausible background manipulations that may be used in attacks involving image spoofing or adversarial content modifications. The dataset and evaluation protocols stress the importance of going beyond object-centric or gross forgery cues, instead focusing on pixel-level localization and robustness to realistic post-processing artifacts such as re-encoding. Models deployed in practice should incorporate evaluation against matched authentic controls to avoid high false positive rates triggered by innocuous processing. Moreover, BG-REAL’s multi-axis OOD testing offers a template for robust performance assessment under distributional shifts that attackers might exploit.

Cite

@article{arxiv2607_26232,
  title={ BG-REAL: A Public Real-Data Anchored Benchmark for Background Manipulation Detection and Localization },
  author={ Bugra Alperen Uluirmak and Rifat Kurban },
  journal={arXiv preprint arXiv:2607.26232},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.26232}
}

Read the full paper

• arXiv:2607.26232 (abstract)
• arXiv:2607.26232 (PDF)