Disentangling Semantic Attention from Structural Bias in the Attention Manifold

Source: arXiv:2607.24017 · Published 2026-07-27 · By Pengkun Jiao, Bin Zhu, Jingjing Chen, Yu-gang Jiang

TL;DR

Multimodal Large Language Models (MLLMs) excel at integrating visual and linguistic inputs but suffer from a subtle structural flaw in their attention mechanism. These models disproportionately attend to semantically uninformative visual tokens or fixed spatial regions, known as "visual attention sinks," leading to diluted semantic signals and frequent visual hallucinations where generated text conflicts with actual visual content. Prior methods tried to fix this by masking or redistributing attention from isolated sink tokens but were computationally expensive and limited in scope. This paper reframes the problem as a pervasive structural bias—query-invariant attention patterns caused by softmax normalization and positional encoding asymmetries—that corrupts the entire attention manifold.

To address this, the authors propose Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play inference-time intervention. SPAR estimates a structural bias template from attention maps marginalized over text tokens and purifies attention by subtracting this bias weighted by visual saliency scores. It then adaptively redistributes the removed attention mass toward confident semantic regions, improving visual signal-to-noise ratio without hurting genuine visual cues. Extensive evaluation on spontaneous hallucination (POPE, CHAIR) and adversarial gaslighting benchmarks (GaslightingBench, MMMU, AI2Diagram, MMBench) demonstrates that SPAR consistently reduces hallucinations and improves model grounding across multiple MLLMs with negligible computational overhead compared to strong baselines like GasEraser. The work advances understanding of attention noise in MLLMs and provides a practical defense against multimodal hallucination.

Key findings

• MLLMs display query-invariant structural bias in text-to-image attention, especially in the first two transformer layers, causing disproportionate attention to fixed spatial regions regardless of input query (Fig 1, 2).
• Structural bias acts as query-invariant noise diluting semantic visual signals and leads to hallucinations by collapsing posterior distributions toward linguistic priors when visual SNR approaches zero (Corollary 1).
• SPAR reduces POPE benchmark accuracy error from 23.31% to 14.8% and improves Precision/Recall/F1 by 4-8 points compared to baseline decoding on LLaVA-v1.5 (Table 1).
• SPAR consistently outperforms or matches state-of-the-art inference-time methods such as AGLA, ONLY, VAR, and GasEraser across hallucination benchmarks, raising post-gaslighting accuracy on GaslightingBench by 3-10 points at minimal latency increase (Table 2).
• Text-side marginalization of attention maps yields better structural bias estimation than image-side marginalization alone, guiding the bias removal step (Table 4).
• Ablation shows saliency guidance, sparsity thresholding, and adaptive probability redistribution are each necessary for maximal SPAR performance, with removal degrading post-negation accuracy by up to 7 points (Table 5).
• SPAR injection limited to top two transformer layers achieves peak hallucination mitigation performance, allowing efficient deployment with minimal inference overhead (Fig 7, Section 5.4).
• Qualitative results demonstrate SPAR sharpens attention maps on semantically relevant image regions, suppressing query-agnostic noise and enhancing true semantic signals (Fig 5, 6).

Threat model

The adversary aims to cause multimodal hallucinations in MLLMs by either exploiting the model's internal structural bias resulting in spontaneous hallucinations, or by providing misleading, adversarial textual prompts (gaslighting) that manipulate the linguistic prior to override visual evidence. The adversary cannot directly modify model weights or attention computation but can control input textual queries and relies on the model's attention susceptibility to discrete or pervasive biases. The defense operates under the assumption that structural bias in attention is fixed and query-invariant.

Methodology — deep read

1. Threat Model & Assumptions: The adversary aims to induce hallucinations in MLLMs either spontaneously (internal model noise/structural bias) or via induced gaslighting attacks that bias the linguistic prior with misleading prompts. The attacker cannot directly modify model weights but can craft adversarial input text vectors. The goal is to prevent the model from ignoring visual evidence due to structural bias in attention.

2. Data: The authors evaluate on multiple standard multimodal hallucination benchmarks, including POPE (object probing), CHAIR (image captioning), GaslightingBench (induced hallucination via negation prompts), and further datasets MMMU, AI2Diagram, and MMBench for generalization. Dataset sizes and splits are inherent from prior works; ground-truth labels include object presence, attribute correctness, and caption fidelity.

3. Architecture/Algorithm: SPAR operates only at inference on pretrained MLLMs (LLaVA-v1.5, LLaVA-NeXT) without retraining. It analyzes text-to-image cross-attention matrices A in early transformer layers (specifically first two layers), which map textual token queries to visual token keys.

The key innovation is decomposing A as A_obs = A_sem + λ * A_struct + ζ, where A_struct is a query-invariant structural bias estimated by marginalizing attention over text tokens, capturing fixed spatial regions disproportionately attended.

SPAR pipeline:

• Compute visual token saliency S_j via normalized feature variances from visual hidden states.
• Marginalize A over text tokens to estimate structural bias template \bar{A}.
• Purify attention: subtract λ*(1 + βS_j)\bar{A_j} from each attention entry, with ReLU to avoid negative mass.
• Compute sparsity ρ_i per text token to estimate confidence γ_i via sigmoid over sparsity offset.
• Calculate removed attention budget Δ_i and redistribute proportional to confidence γ_i to strengthen credible semantic tokens without amplifying noise.

4. Training Regime: As SPAR is training-free, no epochs or hyperparameter optimization beyond grid search for λ, β, and sparsity threshold τ_sparsity (set to 1.0, 1.0, 0.1 respectively). Experiments run on NVIDIA RTX A6000 GPUs.

5. Evaluation Protocol: Multiple evaluation metrics are applied per benchmark, including Precision, Recall, F1 (POPE), Accuracy, CHAIR_S, CHAIR_I for caption hallucination, and robustness to gaslighting measured by pre- and post-negation accuracy. Comparisons are against state-of-the-art inference-time correction methods (AGLA, ONLY, VAR, GasEraser) and baseline multinomial decoding. Ablations analyze marginalization strategy, key module removals, and layer count impact. Cross-validation not explicitly mentioned but reported results use held-out benchmark test splits.

6. Reproducibility: The paper mentions a project page but does not specify code release or frozen weights. The described pipeline operates as a plug-and-play module on existing pretrained LLaVA models without training.

Example End-to-End: For an input image/text pair, SPAR extracts text-to-image attention matrices from the first two transformer layers. It marginalizes these matrices over text tokens to estimate structural bias templates. Feature variances compute saliency scores per visual token. Attention purification subtracts a saliency-weighted structural bias from each attention entry, thresholded by ReLU. Sparsity-based confidence scores guide redistribution of removed attention mass back to strong semantic alignments. This purified attention matrix is used to augment the model’s cross-modal reasoning during generation, resulting in improved visual grounding and reduced hallucination in outputs.

Technical innovations

• Reformulation of visual attention sinks as a pervasive, query-invariant structural bias in the MLLM attention manifold rather than isolated tokens.
• Definition of Semantic-Structural Decoupling hypothesis modeling attention as superposition of latent semantic signals and rigid structural bias, formalized via signal-to-noise ratio in attention space.
• SPAR: a training-free, plug-and-play inference intervention that estimates structural bias templates by marginalizing text-to-image attention over text tokens and purifies attention maps guided by visual feature saliency.
• Adaptive probability redistribution mechanism after bias subtraction to ensure removed attention mass reinforces confident semantic signals rather than amplifying noise.
• Demonstration that injecting SPAR into only the first two transformer layers is sufficient to achieve strong hallucination mitigation with minimal computational overhead.

Datasets

• POPE — unspecified size — public object hallucination benchmark
• CHAIR — unspecified size — public image caption hallucination benchmark
• GaslightingBench — unspecified size — benchmark for induced hallucinations via adversarial prompts
• MMMU — unspecified size — general multimodal understanding benchmark
• AI2Diagram — unspecified size — diagram understanding benchmark
• MMBench — unspecified size — broad multimodal benchmark

Baselines vs proposed

• Regular decoding: POPE Accuracy = 76.69% vs SPAR: 85.20%, F1 score = 79.72 vs 85.42
• AGLA [1]: POPE F1 = 82.61 vs AGLA + SPAR: 83.70
• ONLY [30]: POPE F1 = 82.60 vs ONLY + SPAR: 83.84
• VAR [16]: POPE F1 = 84.61 vs VAR + SPAR: 85.59
• LLaVA-v1.5 GaslightingBench after-negation accuracy: baseline 24.71%, GasEraser 40.95%, SPAR 41.74%
• LLaVA-NeXT GaslightingBench after-negation accuracy: baseline 19.81%, GasEraser 30.58%, SPAR 34.04%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24017.

Fig 1: Visualization of structural attention noise. High

Fig 2: Mean attention maps of (a) text-to-text and (b) text-

Fig 3: SPAR improves robustness to both (a) spontaneous

Fig 4: Illustration of SPAR Pipeline. (a) Text-to-image cross-attention maps are extracted from all attention heads. (b) A

Fig 5: Qualitative visualization of attention purification via SPAR on LLaVA-v1.5. (Left/Middle) Average text-to-image

Fig 6 (page 5).

Fig 7 (page 5).

Fig 8 (page 5).

Limitations

• The structural bias estimation relies on marginalization over text tokens and saliency derived from visual feature variances, which may not generalize to all visual modalities or architectures.
• SPAR is evaluated mainly on two open-source MLLMs (LLaVA-v1.5 and LLaVA-NeXT), so applicability to other multimodal models or larger-scale architectures remains untested.
• No adversarial evaluation targeting the purification mechanism itself—an attacker aware of SPAR might craft attacks to circumvent bias removal.
• The method assumes the availability of meaningful visual saliency signals from visual features, which could be less reliable for highly noisy or out-of-distribution inputs.
• SPAR focuses on softmax attention maps in early Transformer layers; effects on deeper layers or other attention variants are not explored extensively.
• No publicly available code or pretrained models are confirmed released, which may hinder full reproducibility.

Open questions / follow-ons

• How effective is SPAR on other multimodal architectures beyond LLaVA, especially models with different visual encoders or attention mechanisms?
• Can adaptive training approaches complement SPAR's inference-time interventions to further suppress structural bias?
• How resilient is SPAR against adaptive adversarial attacks explicitly designed to exploit or evade structural bias purification?
• What is the impact of SPAR on downstream multimodal tasks beyond hallucination benchmarks, such as reasoning or interactive dialogue?

Why it matters for bot defense

This paper’s analysis and mitigation of structural bias in attention maps have direct implications for bot-defense and CAPTCHA systems that rely on strong multimodal grounding to verify human interaction authenticity. By showing how structural attention bias leads MLLMs to hallucinate or ignore authentic visual input, SPAR provides a practical method to improve the robustness and fidelity of multimodal models without retraining. CAPTCHA designers incorporating vision-language models can adopt similar inference-time purification techniques to ensure that model outputs genuinely reflect the presented visual challenge rather than linguistic shortcuts. Moreover, understanding the latent query-invariant bias can help security researchers detect or prevent manipulation attacks (like gaslighting) that exploit vision-language model weaknesses in the attention manifold. Overall, SPAR offers a lightweight, deployable strategy to reduce hallucination-induced vulnerabilities, strengthening the reliability of multimodal verification in security contexts.

Cite

@article{arxiv2607_24017,
  title={ Disentangling Semantic Attention from Structural Bias in the Attention Manifold },
  author={ Pengkun Jiao and Bin Zhu and Jingjing Chen and Yu-gang Jiang },
  journal={arXiv preprint arXiv:2607.24017},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24017}
}

Read the full paper

• arXiv:2607.24017 (abstract)
• arXiv:2607.24017 (PDF)