Automated Background Swapping for Robustness against Spurious Backgrounds

Source: arXiv:2606.32018 · Published 2026-06-30 · By Cesar Roder, Kajetan Schweighofer

TL;DR

Deep neural network classifiers often rely on spurious correlations embedded in image backgrounds, leading to poor generalization when those correlations break at test time. This paper introduces Automated Background Swapping (AutoBackSwap), a scalable data augmentation technique designed to reduce classifier dependence on spurious background features without requiring samples that break the spurious correlation during training. AutoBackSwap uses a lightweight secondary model trained on a few hundred patch-wise labeled samples to disentangle foregrounds from backgrounds, applies background infilling via generative models or simpler methods to synthesize complete backgrounds, and recombines foregrounds with randomized backgrounds during classifier training. Evaluations on multiple benchmark datasets—Waterbirds, Spawrious, and a novel Spurious Vehicles dataset—demonstrate that AutoBackSwap consistently outperforms prior robustness methods, including those requiring ground-truth segmentations, particularly in challenging scenarios where no minority groups breaking the correlation are available in training. The method shows strong worst-group accuracy improvements of over 13% compared to baselines on Waterbirds without minority groups, and remains effective across diverse tasks and difficulty levels.

Key findings

• On Waterbirds dataset without minority groups, AutoBackSwap achieves 92.2% worst-group accuracy (WGA) vs 79.1% by the closest baseline (Chang et al. [2021]) and 33.4% for standard ERM.
• With minority groups present, AutoBackSwap’s WGA is 93.0%, outperforming all baselines by at least 2 percentage points.
• On the Spawrious many-to-many hardest setting, AutoBackSwap reaches 64.3% WGA compared to 66.7% by Chang et al. and below 50% for other baselines.
• On the novel Spurious Vehicles dataset, standard ERM achieves only 9.5% WGA, while AutoBackSwap with learned detector reaches 87.2%, nearly matching 90.1% with ground-truth masks.
• Training the foreground detector requires only a few hundred patch-wise labeled samples (~287 for Waterbirds) to achieve high accuracy and effective augmentation.
• AutoBackSwap is robust to simpler background infilling methods; AOT-GAN inpainting provides only a modest ~2% WGA gain over scrambles or average color fills.
• Heavy data augmentations like mixup and CutMix degrade performance, indicating that certain augmentations may reinforce background biases.
• Zero-shot foundation models like CLIP and SAM3 perform worse than task-specific methods in settings with strong spurious correlations.

Threat model

The threat model considers the natural biases and spurious correlations present in training data between target labels and background attributes, which can cause trained classifiers to rely on non-causal cues. The adversary is implicit: these biases manifest as shortcuts the model learns. There are no explicit adversarial modifications or attacks assumed; rather the challenge is to train models robust to distribution shifts where these correlations break during inference.

Methodology — deep read

AutoBackSwap targets the problem of spurious background correlations in image classification, where the background attribute 'a' correlates with class label 'y' but is not causally related. The assumption is that foreground content causally determines the label, while the background determines spurious attribute 'a'. The adversary is the natural dataset bias inducing this spurious correlation; no adversarial manipulations are considered.

Data consists of standard classification datasets with images (x), labels (y), and spurious attributes (a). Training sets may or may not contain samples that break the correlation between y and a. An auxiliary dataset with patch-wise binary masks (z) indicating foreground vs. background is used to train the detetctor model. The size of this auxiliary dataset is small (a few hundred labeled samples) relative to the full dataset; labeling focuses on patch-level binary masks at tuned resolutions to balance annotation cost and quality.

The key components are:

1. Foreground/Background Detector: A dedicated patch-wise classifier (similar architecture to the main classifier, e.g., ResNet50) trained on the auxiliary dataset to predict a binary mask m for each input image.
2. Background Generator: A background infilling model, primarily AOT-GAN, which takes the background portion of the image (background defined as x ⊙ (1-m)) with masked-out foreground and fills in realistic background content to produce a complete background image.
3. Recomposition: During training, foreground crops x_fg are combined with randomly sampled infilled backgrounds from different spurious attribute groups, breaking the spurious correlation in augmented data.

Training proceeds by constructing two datasets from the original data using the detector: foregounds with labels (x_fg,y) and backgrounds with spurious attribute labels (x_bg,a). For each training step, a foreground is sampled uniformly by class, a spurious attribute is sampled uniformly, a background with that attribute is sampled from B, and foreground and background are recombined to form augmented images that train the classifier c via ERM.

Foreground augmentations like resizing, rotation, and translation are applied before recomposition to further enhance robustness. When spurious attributes are unknown, background sampling is approximated by sampling based on class labels assuming strong correlation between class and background.

Experiments use pretrained ResNet50 models with standard hyperparameters and early stopping on validation accuracy. Evaluation metrics include overall accuracy and worst-group accuracy (minimum group accuracy over the defined y,a groups).

Multiple datasets are evaluated: Waterbirds (binary classes with water vs land backgrounds), Spawrious (four dog breeds with six backgrounds), and Spurious Vehicles (four vehicle types in four contexts). Ablations study effects of auxiliary dataset size and patch resolution, foreground mask label quality, background infilling methods, and sampling strategies for recomposition.

The approach is reproducible given patch-wise labeled auxiliary data and relies on accessible generative inpainting models or simple infilling heuristics. Code and datasets were not explicitly marked as public in the source. Overall, AutoBackSwap is a practical augmentation-based strategy that can be layered on standard classifiers without requiring group annotations or ground-truth segmentations for the full dataset.

Technical innovations

• Introduction of a lightweight patch-wise foreground/background detector trained on a small auxiliary dataset with patch-level masks to disentangle image components.
• Use of generative background infilling to synthesize complete backgrounds after foreground masking, enabling full recomposition of novel image pairs.
• A stochastic recomposition training procedure that combines foregrounds and randomized infilled backgrounds to break spurious correlations purely via data augmentation.
• Demonstration that effective robustness against spurious backgrounds can be obtained even without any training samples that break the spurious correlation.
• Empirical finding that simpler infilling methods (e.g., pixel or patch scrambling) suffice for strong robustness gains, lowering the barrier for practical deployment.

Datasets

• Waterbirds — ~4,700 images — public benchmark with synthetic backgrounds [Sagawa et al., 2020]
• Spawrious — several thousand images of four dog breeds on six background types — synthetic, generated with text-to-image diffusion model [Lynch et al., 2023]
• Spurious Vehicles — synthetic dataset introduced in paper, four vehicle classes in four contextual backgrounds, details and splits provided by authors

Baselines vs proposed

• ERM on Waterbirds (no minority groups): WGA=33.4%, Acc=68.4% vs AutoBackSwap WGA=92.2%, Acc=95.1%
• Chang et al. [2021] on Waterbirds (no minority groups): WGA=79.1%, Acc=91.4% vs AutoBackSwap WGA=92.2%, Acc=95.1%
• DFR on Waterbirds (no minority groups): WGA=15.1%, Acc=60.1% vs AutoBackSwap WGA=92.2%, Acc=95.1%
• AutoBackSwap vs Chang et al. [2021] on Spawrious many-to-many hard: WGA=64.3% vs 66.7%, Acc=85.7% vs 85.7%
• AutoBackSwap vs ERM on Spurious Vehicles (m2m): WGA=87.2% vs 9.5%, Acc=95.3% vs 37.9%
• AutoBackSwap with GT segmentation on Spurious Vehicles: WGA=90.1%, Acc=96.3% vs learned detector WGA=87.2%, Acc=95.3%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.32018.

Fig 5: Illustration of applying AutoBackSwap to a dataset of four images, given an already

Fig 6: The Waterbirds dataset has four groups that split up into minority (landbirds on water,

Fig 7: Waterbirds exam-

Fig 9: Detailed configurations of Spawrious datasets. One-to-One: Train set uses a dominant and

Fig 8: Spawrious exam-

Fig 6 (page 15).

Fig 7 (page 15).

Fig 8 (page 15).

Limitations

• Dependence on small auxiliary patch-wise labeled dataset; labeling cost and accuracy tradeoffs remain a factor.
• Quality of foreground/background separation heavily influences ultimate performance; learned detector underperforms ground-truth segments.
• Background infilling methods require domain-appropriate models; though simple methods work, visual quality may vary.
• Experiments do not test explicit adversarial attacks or more complex spurious attribute shifts beyond the datasets considered.
• Evaluation mainly on synthetic or semi-synthetic benchmarks combining foregrounds and backgrounds; real-world generalization requires further validation.
• Zero-shot and foundation model evaluation limited to few models; future wider tests needed.

Open questions / follow-ons

• Can AutoBackSwap be adapted or combined with stronger domain generalization or invariant risk minimization methods for further robustness?
• What are the tradeoffs and annotation strategies for scaling auxiliary patch-wise labeling to real-world large datasets with complex backgrounds?
• How do different generative inpainting models or diffusion methods affect robustness and computational cost in diverse domains?
• Can foreground/background disentanglement be improved via self-supervised or weakly supervised approaches reducing dependency on annotation?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, AutoBackSwap offers an approach to mitigate the risk that visual classifiers rely on spurious, background-based cues that can be manipulated or do not generalize well in deployment environments. Incorporating automated background swapping augmentation during training can help produce more robust image-based classifiers that focus on meaningful foreground objects, reducing the chance bots exploit background shortcuts for evasion. The method’s low annotation overhead and compatibility with standard classification pipelines make it practical for training robust visual recognition components used in CAPTCHAs or bot detection systems, especially where backgrounds vary widely or adversaries might manipulate context. However, engineers should be aware that quality foreground/background separation is critical, and that background infilling does not need to be perfect—simple methods suffice. Overall, AutoBackSwap provides a data augmentation framework to enhance the causal robustness of visual models crucial for trustworthy bot-defense.

Cite

@article{arxiv2606_32018,
  title={ Automated Background Swapping for Robustness against Spurious Backgrounds },
  author={ Cesar Roder and Kajetan Schweighofer },
  journal={arXiv preprint arXiv:2606.32018},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.32018}
}

Read the full paper

• arXiv:2606.32018 (abstract)
• arXiv:2606.32018 (PDF)