InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation

Source: arXiv:2607.19324 · Published 2026-07-21 · By Khawar Islam, Arif Mahmood, Xin Jin, Naveed Akhtar

TL;DR

InstructMixup addresses key limitations of traditional mixup-based data augmentation methods, which typically combine content from multiple images and often disrupt semantic consistency. Instead, it generates challenging yet label-consistent augmentations entirely within a single image by extracting multi-scale salient patches and performing instruction-guided generative edits on these patches. These edited patches are then reintegrated into non-salient regions of the same image, blending fractal texture selectively to increase structural complexity while preserving key semantics. This approach avoids high runtime costs by precomputing generative edits offline and maintains original hard labels since no cross-image mixing is done.

The method further benefits from randomly mixing its augmentation mode with standard methods like Mixup, CutMix, and ResizeMix to enhance diversity. The authors provide a novel theoretical analysis using a second-order vicinal risk expansion, interpreting the method as enforcing invariance to structured transformations (generative edit, rotation, blur) and penalizing curvature along salient, perturbed directions via fractal injection. Exhaustive evaluations across seven diverse datasets and several backbone architectures (CNNs, vision transformers, vision-language models) demonstrate InstructMixup consistently outperforms nine strong baselines on clean accuracy, robustness to corruption and occlusion, calibration, transfer learning, and few-shot tasks.

Key findings

• InstructMixup surpasses nine competing augmentation methods including Mixup, CutMix, PuzzleMix, AutoMix, and AdAutoMix on seven benchmarks covering coarse- and fine-grained classification, robustness, calibration, and self-supervised learning.
• The generative edit step is computed offline and cached, adding negligible training overhead compared to prior saliency-guided mixup methods that require online optimization and saliency computations.
• Restricting fractal texture injection to salient patches (AdaFrac) avoids the distribution shifts caused by global fractal blending methods like DiffuseMix, improving robustness without sacrificing data fidelity.
• Randomly mixing augmentation modes (InstructMixup, CutMix, Mixup, ResizeMix) rather than committing to a single mixing strategy significantly boosts generalization.
• A second-order vicinal risk approximation shows InstructMixup simultaneously enforces invariance to generative edits and spatial transforms and penalizes loss curvature locally along salient image directions (Fig 5, Sec IV).
• The instruction-guided generative editing ensures label consistency via a pretrained verifier network filtering out disallowed edits (Eq 6).
• Evaluation spans multiple backbones (ResNet-18/34/50, ResNeXt-50, Swin-T, ViT-B, ConvNeXt-T, CLIP) demonstrating architecture-agnostic effectiveness.
• Improvements hold under realistic distribution shifts: occlusion, corruption (ImageNet-C, CIFAR-100-C), and few-shot transfer (Oxford Flowers-102, Pascal VOC detection).

Threat model

Not applicable; the paper focuses on improving model generalization and robustness through augmentation rather than defending against adversarial or malicious attacks.

Methodology — deep read

Threat model and assumptions: InstructMixup assumes a standard supervised visual classification scenario; the augmentation introduces label-consistent transformations within each training sample to improve model generalization and robustness. The adversary is not explicitly modeled, as the focus is on improving model invariance, not security against targeted attacks.

Data: The authors evaluated on seven standard datasets covering coarse- and fine-grained classification (CIFAR-100, Tiny-ImageNet, ImageNet-1K, CUB-200-2011, FGVC-Aircraft, Stanford-Cars, Oxford Flowers-102), robustness benchmarks (CIFAR-100-C, ImageNet-C), and Pascal VOC for detection. The datasets contain thousands to millions of labeled images, with standard train/test splits and standard preprocessing (random crop, flip for CIFAR-100; ImageNet standard resizing).

Architecture/Algorithm: InstructMixup operates per sample by first computing a spectral-residual saliency map (Φ) to identify multi-scale salient patches at two scales (H/2 x W/2 and H/4 x W/4). Candidate patches are randomly sampled and accepted if their salient area ratio surpasses a random threshold τ in [0.5,1.0], ensuring patch variability and preventing overfitting to single granularity.

Each accepted patch qj is edited by an offline instruction-guided generative model (E(qj, mask)) that adapts appearance within salient regions while preserving semantics. Edits are cached and only retained if a frozen classifier verifier V predicts the same class, enforcing label consistency.

The edited patch is blended with a fractal texture from a preconstructed fractal library Z via qb_j = β z + (1-β) qe_j (β=0.20). Subsequently, a dual-stream transform applies a rotation R (angle randomly sampled from Uniform(-30°,30°)) to the salient stream and Gaussian blur to the non-salient stream, then resized back to full image resolution.

All patches pj are weighted equally and recomposed with the original image via weighted sum controlled by a random global blend γ ~ Uniform(0,1). The original hard label yi is preserved in all cases.

Training regime: For CIFAR-100, ResNet-18 and ResNeXt-50 models were trained for 200 epochs using SGD with momentum 0.9, weight decay 1e-4, batch size 100, initial learning rate 0.1 decayed by cosine annealing schedule. ImageNet-1K trained for 100 epochs with batch size 256 under similar hyperparameters. Fine-grained datasets fine-tuned from pretrained weights for 200 epochs with batch size 16, learning rate 0.001 decayed cosine-wise. Self-supervised learning with MoCo v2 and SimSiam followed established protocols, applying InstructMixup to one of the two views.

Evaluation protocol: Metrics included top-1 classification accuracy on test splits, robustness under corruption (IN-C, CIFAR-100-C), occlusion, calibration error, transfer learning tasks including few-shot classification, and object detection mAP on Pascal VOC. Strong baselines (Mixup, CutMix, PuzzleMix, AutoMix, AdAutoMix, etc.) were reimplemented or compared directly per the OpenMixup framework. Ablations on fractal blend strength β, mixing mode combinations, and generative edit quality were conducted. Statistical tests were not explicitly reported.

Reproducibility: The generative edits require a pretrained instruction-guided image editor (not named), and a spectral residual saliency detector is used. Edits are cached offline before augmentation. The fractal library Z is constructed once. Code release is mentioned in the original OpenMixup framework but no explicit code or weights for InstructMixup are included in the summary. The datasets used are all public standard benchmarks.

Technical innovations

• Instruction-guided generative editing of multi-scale salient patches within the same image to introduce realistic appearance variations while preserving semantic label consistency.
• Selective fractal texture blending constrained to salient patches (AdaFrac) rather than globally, avoiding out-of-distribution shifts caused by full-image fractal overlays.
• Randomized acceptance threshold for saliency mask during patch extraction to yield diverse scale-aware patch augmentations within the image.
• Theoretical second-order vicinal risk approximation decomposing the augmentation objective into an invariance term enforcing robustness to generative edits and a saliency-localized Hessian penalty from fractal perturbations.

Datasets

• CIFAR-100 — 60,000 images (50,000 train, 10,000 test) — public benchmark
• Tiny-ImageNet — 100,000 images (train/val/test splits) — public benchmark
• ImageNet-1K — ∼1.28 million training, 50,000 validation images over 1,000 classes — public benchmark
• CUB-200-2011 — 11,788 images (5,994 train, 5,794 test) — public benchmark
• FGVC-Aircraft — ∼10,000 images (train/val/test) — public benchmark
• Stanford-Cars — 16,185 images (8,144 train, 8,041 test) — public benchmark
• Oxford Flowers-102 — 8,189 images across 102 classes — public benchmark
• Pascal VOC — ∼15,000 images for object detection tasks — public benchmark
• CIFAR-100-C and ImageNet-C — benchmark corruption datasets derived from CIFAR-100 and ImageNet

Baselines vs proposed

• Mixup: CIFAR-100 top-1 accuracy ≈ 74.4% vs InstructMixup: 77.2%
• CutMix: CIFAR-100 top-1 accuracy ≈ 76.5% vs InstructMixup: 77.2%
• PuzzleMix: CIFAR-100 top-1 accuracy ≈ 75.8% vs InstructMixup: 77.2%
• AutoMix: CIFAR-100 top-1 accuracy ≈ 75.9% vs InstructMixup: 77.2%
• AdAutoMix: CIFAR-100 top-1 accuracy ≈ 76.4% vs InstructMixup: 77.2%
• S2-FracMix preliminary variant: lower performance than InstructMixup by ~1.5-2% top-1 accuracy across datasets
• Robustness on ImageNet-C corruption benchmark: InstructMixup reduces error rates by more than 3% relative to strongest baseline

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.19324.

Fig 1: Representative augmentation samples created by different mixup methods. Each competing method constructs its sample from a source and a target

Fig 2: InstructMixup overview. In the InstructMixup mode, the spectral-residual detector computes a saliency map and selects salient patches at two scales.

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• The generative editing relies on pretrained instruction-guided image editors whose training data or domain overlap with target datasets is unclear, possibly limiting applicability to out-of-domain images.
• The label consistency check using a pretrained fixed verifier network may not perfectly guarantee semantic correctness for all generative edits; no adversarial or human evaluation reported.
• Computational overhead for precomputing generative edits and constructing fractal library is not quantified; requires substantial offline preprocessing resources.
• Robustness and generalization tests are limited to standard corruptions and occlusions; no adversarial robustness or strong distribution shift evaluations included.
• Multi-mode mixing strategy introduces randomness which may complicate reproducibility and interpretation of individual augmentation effects.
• The second-order vicinal risk approximation assumes smoothness and zero-mean perturbations; real generative edits might violate these assumptions in practice.

Open questions / follow-ons

• How well does instruction-guided generative editing transfer to domains with less well-defined salient regions or highly diverse object classes?
• Can the framework be extended to video or temporal data preserving temporal consistency in generative edits?
• How would adversarial robustness change if generative edits are optimized to maximize or minimize model confidence?
• What are the trade-offs between offline cost of caching generative edits versus online augmentation for large-scale production deployments?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, InstructMixup offers a sophisticated augmentation strategy that enhances model robustness to natural variations in visual data without compromising the semantic integrity of critical features. This is particularly important for CAPTCHA systems that rely on discriminative features for human/bot classification under noisy or adversarial conditions.

The instruction-guided generative editing and fractal-based perturbations simulate realistic appearance and texture changes localized within salient object regions, which likely improves resilience to automated solvers attempting to exploit invariant cues or background noise. The offline caching of generative edits also suggests that such augmentations can enhance training pipelines without adding runtime latency, valuable for scalable CAPTCHA system retraining. Overall, adopting such augmentation approaches can help improve the robustness and generalization of models that underpin bot-detection and CAPTCHA solvers against distribution shifts and novel bot strategies.

Cite

@article{arxiv2607_19324,
  title={ InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation },
  author={ Khawar Islam and Arif Mahmood and Xin Jin and Naveed Akhtar },
  journal={arXiv preprint arXiv:2607.19324},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.19324}
}

Read the full paper

• arXiv:2607.19324 (abstract)
• arXiv:2607.19324 (PDF)