MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents
Source: arXiv:2606.20717 · Published 2026-06-16 · By Xuelong Dai, Jianyu Ma, Boyang Ma, Biwei Yan, Yijun Yang, Yue Zhang
TL;DR
This paper addresses a realistic and practical security challenge in multimodal large language model (MLLM)-based web agents that automate browser tasks by observing webpage screenshots. Specifically, it studies whether an adversary limited to controlling only a semantically legitimate, spatially constrained visual region (e.g., an ad slot) can hijack the agent's next action prediction without direct access to the webpage DOM or agent internal prompts. The authors introduce MIRAGE, a novel visual prompt injection framework that uses diffusion models to generate perceptually benign adversarial images confined strictly within attacker-controlled boundaries. They combine curvature-aware adversarial diffusion guidance with sparse pixel-level residual perturbations constrained to dark pixels to achieve high attack efficacy and stealth.
MIRAGE is evaluated extensively against open-source MLLM web agents such as SeeAct and OpenClaw, powered by multiple multimodal LLM backends including LLaVA, Phi, Llama, and Qwen. Results demonstrate MIRAGE consistently achieves state-of-the-art attack success rates (ASR of 70-98%) on a real-world Mind2Web dataset subset, outperforming existing screenshot-level and text-based attack baselines. The attack also maintains strong visual stealthiness by optimizing perceptual similarity (lowest LPIPS among baselines) and minimizing conspicuous perturbations. Ablation studies validate the importance of the diffusion-guided semantic latent manipulation and sparse dark-pixel perturbations. This work substantially extends vulnerability detection capabilities for MLLM-driven web agents restricted to bounded attacker-controlled regions, highlighting a new practical attack vector.
Key findings
- MIRAGE achieves 90.4%-97.2% attack success rate (ASR2) on SeeAct across multiple MLLM backends, surpassing prior methods like WebInject* (highest baseline ASR2 75.4%) and popup attacks (max ASR2 27.0%)
- On OpenClaw with Qwen2.5-VL backend, MIRAGE reaches 98.5% ASR, outperforming the best baseline at 44.2% ASR
- Diffusion-guided latent manipulation combined with sparse dark-pixel residual perturbations yields highest stealth, with LPIPS of 6.8x10^-2 outperforming WebInject* (27.5x10^-2) and popup attacks (9.3x10^-2)
- MIRAGE’s composite adversarial patch is strictly confined within attacker-controlled 300x300 pixel bounding boxes, preserving the integrity of site-owned webpage elements
- Curvature-aware adversarial diffusion guidance increases attack success by avoiding local minima, confirmed via ablation that higher curvature loss weight improves ASR but can degrade visual quality (Fig. 2)
- Sparse dark-pixel residual perturbations focused on the lowest 30% luminance pixels improve visual stealth while enabling precise next-action hijacking
- The attack requires fewer iterations and less perturbation than unrestricted full-screen baseline attacks, with acceptable average running time around 60 minutes per task
- Text-based attacks show significantly degraded performance and are impractical within the bounded-region threat model
Threat model
The adversary is an unprivileged third party operating on a trusted web platform, controlling only a spatially bounded, semantically legitimate visual region embedded in the webpage (e.g., an ad slot). They cannot modify the global DOM, browser execution, or system prompts, nor access internal memory or APIs of the MLLM agent. During attack generation, white-box gradient access to the open-source MLLM models is permitted, but at runtime the model parameters are fixed.
Methodology — deep read
The threat model assumes a trusted web platform hosting screenshot-based MLLM web agents (e.g., SeeAct, OpenClaw). The adversary is an unprivileged third party controlling only a semantically legitimate, spatially bounded visual region (roughly 300x300 pixels) such as an advertisement or merchant widget embedded on the page. The adversary cannot modify the webpage DOM, browser execution, or agent system prompts, and the rest of the screenshot remains unaltered.
The attacker’s objective is to inject a localized visual prompt into the attacker-controlled region to hijack the agent’s next browser action prediction during the MLLM’s action generation phase. The next action is textual output subsequently grounded to a UI element. The attacker controls full white-box gradient access to the target open-source MLLMs during adversarial patch generation, but at runtime, the model parameters remain fixed.
Data provenance: The evaluation uses the Mind2Web dataset (Deng et al., 2024), containing over 2000 web navigation tasks across 137 websites and 31 domains. The evaluation subset only includes 163 tasks requiring CLICK or SELECT actions, with full screenshots, user queries, histories, and system prompts retained from the original split. Attacker-controlled mask regions are manually selected as legitimate bounded regions on these real website screenshots.
Architecture: The core innovation is MIRAGE, which combines semantic-level latent manipulation with sparse pixel-level perturbations within the attacker mask. The semantic manipulation uses latent diffusion model sampling on masked inputs, performing image-to-image translation by partially applying forward diffusion noise to the latent representation of the patch, then running a reverse DDIM sampling process guided by gradients of an adversarial loss.
The adversarial loss Ltask maximizes likelihood of attacker-chosen target textual next actions generated by the web agent's action generation module given the composite screenshot with adversarial patch. To improve search and escape local minima, a curvature-aware loss Lcurve encourages large second-order variations in the DDIM noise prediction trajectory during sampling. These two losses compete during iterative latent updates.
Complementing semantic latent synthesis, a sparse residual pixel-level perturbation δ restricted to the darkest 30% luminance pixels in the masked patch is optimized with projected gradient descent to refine the perturbation. Both diffusion latent and residual pixels are composited differentiably with the original screenshot outside the mask.
Training: The diffusion latent is initialized from the encoded attacker-region of the benign screenshot. Partial forward diffusion applies noise up to 30% of T=100 denoising steps. The reverse DDIM sampling runs for T=30 timesteps. The latent is updated iteratively for K=5 outer PGD iterations, balancing Ltask and Lcurve with weight λ1=0.01. Residual perturbations use learning rate 2/255 constrained in a 16/255 perturbation L∞ ball. The whole pipeline is optimized jointly using white-box gradients of the target multi-modal LLM web agents over their open-source backends (LLaVA, Phi, Llama, Qwen).
Evaluation protocol: Attack success rates (ASR1 for next action generation alignment, ASR2 for full grounding match) are evaluated on the Mind2Web subset with 5 repeat runs. Baselines include reproductions of WebInject*, EIA, Popup Attack, and text-based attacks Naive Attack and Context Ignoring. Stealth is quantified using LPIPS perceptual similarity and Total Variation metrics. Ablations vary diffusion sampling timesteps T, update steps K, and curvature loss weight λ1. Statistical significance testing is not explicitly reported.
A concrete example: Starting from a user screenshot with an attacker-controlled ad slot, MIRAGE encodes the mask region, applies partial forward diffusion noise to the latent, then applies iterative reverse DDIM sampling modifying the latent guided by gradients maximizing likelihood of the attacker-specified next action (e.g., 'Click Add to Cart'). Sparse residual pixel perturbations restricted to dark pixels are updated concurrently. The adversarial patch latent and residuals are decoded and composited into the original screenshot. The composite is evaluated by the MLLM web agent, showing hijacked next-action prediction with stealthy visual appearance preserved.
Reproducibility: Code and exact model weights are not publicly released by the authors. The Mind2Web dataset is publicly available. The baselines are reproduced or implemented by the authors. Details on hyperparameters and models are thoroughly described.
Technical innovations
- Formulation of a strict bounded-region visual prompt injection threat model for trusted web platforms allowing attacker control over only a semantically legitimate webpage region.
- Integration of diffusion-model guided latent semantic manipulation with sparse dark-pixel residual perturbations for precise targeted next-action hijacking under strict spatial constraints.
- Curvature-aware adversarial diffusion sampling that encourages DDIM latent trajectories to diverge from benign manifolds, escaping local minima and enhancing attack strength.
- A fully differentiable compositing pipeline jointly optimizing latent diffusion outputs and sparse residual perturbations confined to attacker masks.
Datasets
- Mind2Web Multimodal Subset — 163 tasks — publicly released real-world webpage screenshots with CLICK/SELECT actions
Baselines vs proposed
- SeeAct-LLaVA ASR2: WebInject* = 75.4% vs MIRAGE = 90.4%
- SeeAct-Llama ASR2: WebInject* = 18.4% vs MIRAGE = 70.2%
- SeeAct-Phi ASR2: WebInject* = 17.2% vs MIRAGE = 97.2%
- SeeAct-Qwen ASR2: WebInject* = 41.7% vs MIRAGE = 95.6%
- OpenClaw-Qwen ASR: WebInject* = 25.8% vs MIRAGE = 98.5%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20717.
Fig 1: Illustration of MIRAGE. A trusted webpage contains a bounded region controlled by a third-party
Fig 2 (page 3).
Fig 3 (page 3).
Fig 4 (page 3).
Fig 5 (page 3).
Fig 6 (page 3).
Fig 4: The cross-attention heatmaps from the clean screenshot and MIRAGE screenshot. The red box
Fig 5: An attack screenshot with Naive Attack.
Limitations
- The approach currently assumes white-box access to the target open-source MLLM gradients limiting applicability to closed-source or black-box proprietary web agents.
- Relatively high computational overhead due to iterative PGD and diffusion latent sampling increases attack generation time compared to purely gradient-based methods.
- Visual stealth relies partly on manually selected attacker-controlled mask regions and luminance thresholds; adaptability and automation of mask selection were not explored.
- No evaluation under distributional shift or against defense mechanisms; robustness to detection or adversarial training is untested.
- No statistical significance testing reported for result variability beyond 5-run averaging.
Open questions / follow-ons
- How effective is MIRAGE against closed-source or black-box MLLM web agents where gradient access is unavailable?
- Can the framework be adapted to automate mask selection and perturbation without manual region annotation?
- What defenses or detection strategies can mitigate such stealthy localized visual prompt injections?
- How does MIRAGE perform under perturbation constraints simulating real-time or mobile computational budgets?
Why it matters for bot defense
This work demonstrates a novel and realistic visual attack vector targeting multimodal LLM-based web agents used in browser automation, which may overlap with bot-defense mechanisms that rely on visual content interpretation. Bot-defense engineers should be aware that even visually legitimate third-party controlled webpage regions can stealthily and precisely hijack next-action predictions without obvious perturbations or DOM tampering.
Practitioners designing CAPTCHAs or visual bot-detection approaches for MLLM agents could leverage insights from MIRAGE’s curvature-aware diffusion sampling and sparse perturbation techniques to better understand how adversaries might craft localized visual content that evades detection. The attack’s stealth techniques show that simple pixel-noise-based methods may be insufficient for defense. Defensive strategies might require analyzing localized attention patterns or incorporating robust multi-modal adversarial training inspired by these findings.
Cite
@article{arxiv2606_20717,
title={ MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents },
author={ Xuelong Dai and Jianyu Ma and Boyang Ma and Biwei Yan and Yijun Yang and Yue Zhang },
journal={arXiv preprint arXiv:2606.20717},
year={ 2026 },
url={https://arxiv.org/abs/2606.20717}
}