Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

Source: arXiv:2607.02484 · Published 2026-07-02 · By Xuehui Wang, Xuankun Yang, Wei Shen

TL;DR

This paper addresses key challenges in visual token pruning for vision-language models (VLMs), which aim to accelerate inference by selecting a smaller subset of informative image tokens (patches). Prior pruning approaches degrade under dense textual instructions and fine-grained queries due to two fundamental bottlenecks: (1) textual noise dispersion in dense cross-modal scoring where functional words produce diffuse activations that drown out meaningful visual-text alignment signals, and (2) feature fragmentation and redundancy from naive Top-K token selection that over-focuses on local discriminative regions and ignores holistic visual context. To tackle these, the authors propose Entropy-Aware Dense Pruning (EADP), which uses statistical entropy to filter noisy text tokens for fine-grained relevance scoring, then formulates token selection as a spatially-aware submodular maximization problem that ensures diverse, non-redundant visual coverage. Extensive experiments on multiple VLM architectures (LLaVA, Qwen2.5/3-VL) and over a dozen multimodal benchmarks demonstrate that EADP robustly preserves critical textual-visual cues under tight token budgets, consistently outperforming strong prior pruning methods while achieving favorable accuracy-efficiency trade-offs. The method also generalizes well across input resolutions and heterogeneous VLM backbones, retaining fine-grained information needed for compositional reasoning and document tasks.

Key findings

• Entropy-guided text token filtering removes dispersed textual noise, improving dense cross-modal relevance accuracy over naive dense aggregation without NLP parsers.
• EADP’s fusion of dense low-entropy scores with global EOS token yields instruction relevance maps that capture both fine-grained and global semantic cues.
• Submodular maximization token selection, combined with Gaussian spatial smoothing and score polarization, fully covers semantic regions and avoids local redundancy seen in Top-K selection.
• On LLaVA-1.5-7B, EADP retains 128 tokens (77.8% pruning) with 63.5 average accuracy, surpassing CDPruner’s 63.2 and DivPrune’s 62.5 (Tab. 1).
• At extreme pruning on LLaVA-1.5-7B (32 tokens, 94.4% pruning), EADP outperforms DivPrune by 2.5 points average accuracy.
• On high-res LLaVA-NeXT-7B (2880 tokens), EADP with 640 tokens matches unpruned accuracy at 66.3 average, exceeding other methods by at least 0.3 points (Tab. 2).
• On Qwen2.5-VL, EADP maintains 74.3 average at 256 tokens and 68.4 average at 128 tokens, outperforming DivPrune, HiPrune, and CDPruner (Tab. 3).
• EADP improves document reasoning benchmarks (DocVQA, InfoVQA) substantially, e.g. +7.0 and +9.8 average over DivPrune and CDPruner at 256 tokens on DocVQA.

Threat model

Not applicable as the paper studies visual token pruning for efficient VLM inference rather than adversarial security. The focus is on maintaining critical visual-text alignment under tight token budgets assuming standard multimodal inputs without an active adversary.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly defined as this is a model compression task rather than an adversarial attack. The focus is on preserving critical visual-text alignment cues under dense and fine-grained instructions while pruning visual tokens. It is assumed that a VLM encodes the image into N visual tokens and the text prompt into M tokens plus a global EOS token. The challenge is that the dense instructions contain functional words/punctuation contributing textual noise that corrupts naive dense cross-modal scoring.

2. Data: The method is evaluated on standard multimodal vision-language benchmarks including VQAv2, GQA, VizWiz, ScienceQA-IMG, etc., with various token budgets. Across LLaVA and Qwen VLM families, this covers a mix of fine-grained object queries, compositional reasoning, and document visual question answering. Evaluation splits follow official protocols.

3. Architecture / Algorithm: Given visual tokens V (N patches) projected into CLIP embedding space and text tokens T from CLIP text encoder, the authors compute dense cross-modal similarity matrices (Eq. 3). To quantify textual noise, they convert each text token's similarity distribution over image tokens into a probability distribution using softmax (Eq. 4) and compute the statistical entropy (Eq. 5) for each text token. Tokens above the q-th entropy quantile are filtered out, keeping only low-entropy candidates weighted by a softmax over negative entropy (Eq. 7). This dispersion-aware dense guidance score SD is aggregated across retained tokens (Eq. 8) and fused with the global EOS token similarity score SG to yield instruction relevance SI (Eq. 9–10).

To resolve feature fragmentation, the 1D relevance vector SI is reshaped to a 2D spatial map and smoothed with a 3x3 Gaussian kernel to propagate local context (Eq. 11). The smoothed scores are polarized with an exponential function (β > 1) to amplify peaks and suppress background (Eq. 12).

Token selection is formulated as a facility location submodular maximization problem (Eq. 13) that optimizes the sum over all tokens of the maximum similarity between token and selected subset weighted by polarized instruction relevance. This formulation explicitly encourages holistic coverage and penalizes redundant local maxima. A greedy algorithm with cached marginal gains is used for efficient approximate optimization.

4. Training Regime: The pruning module is plug-and-play without end-to-end training of the VLM. No additional training beyond standard pretraining of underlying models is described. Hyperparameters like entropy quantile q, temperature τ, polarization β are tuned via validation.

5. Evaluation Protocol: Metrics are average accuracy over multiple VQA benchmarks under varying pruning budgets (e.g. 77.8%, 88.9%, 94.4% token reduction). Baselines include recent state-of-the-art pruning methods like CDPruner, DivPrune, HiPrune, SparseVLM, etc. Experiments cover both standard (e.g., 224x224) and high resolution (672x672, 1008x1008) inputs, with evaluation on held-out datasets. Ablations study the impact of entropy filtering, submodular selection vs Top-K, and smoothing/polarization.

6. Reproducibility: Code is publicly released (https://github.com/SJTU-DeepVisionLab/EADP) enabling reproduction. Dataset splits are standard public VQA and multimodal benchmarks. Pretrained VLM weights are leveraged from open sources. Detailed hyperparameter choices and implementation details appear in supplemental.

Concrete Example: For an image and prompt "Is there an oven in the image?", EADP first computes dense similarity between each text token excluding EOS and all visual tokens. Text tokens like "oven" have low entropy indicating focused attention, while function words like "is" have high entropy and are filtered out. The retained low-entropy tokens' similarities are weighted and aggregated to produce a dense relevance score SD per visual token. This is fused with the global EOS token score and normalized to SI. SI is reshaped to spatial layout and smoothed via Gaussian filter, then polarized to sharpen peaks. Finally, tokens are selected by greedy maximization of the submodular facility location objective ensuring broad coverage of oven parts rather than concentrating on one region, yielding a compact and informative token subset for downstream VLM reasoning.

Technical innovations

• Introduction of entropy-guided denoising to quantify and filter dispersed textual noise from functional words in dense cross-modal similarity scoring without relying on external NLP parsing.
• Fusion of entropy-aware dense relevance scores with global EOS guidance for robust instruction relevance maps combining fine-grained precision with global semantic stability.
• Formulation of visual token selection as a facility location submodular maximization problem incorporating spatial smoothing and score polarization to guarantee holistic, non-redundant token subset selection.
• Use of a lightweight Gaussian convolution and exponential score polarization to incorporate spatial continuity and amplify core visual features prior to token selection.

Datasets

• VQAv2 — ~265K images — public
• GQA — ~113K images — public
• VizWiz — ~31K images — public
• ScienceQA-IMG — size not specified — public
• TextVQA — ~28K images — public
• POPE — size not specified — public
• MME — size not specified — public
• MMBench, MMBench-CN, MM-Vet — size not specified — public
• ChartQA, AI2D, OCRBench, HallBench, DocVQA, InfoVQA — sizes not specified — public or benchmark-specific

Baselines vs proposed

• CDPruner(NIPS 2025): avg accuracy at 128 tokens = 63.2 vs EADP = 63.5 (LLaVA-1.5-7B)
• DivPrune(CVPR 2025): avg accuracy at 128 tokens = 62.5 vs EADP = 63.5 (LLaVA-1.5-7B)
• CDPruner(NIPS 2025): avg accuracy at 64 tokens = 61.9 vs EADP = 62.2 (LLaVA-1.5-7B)
• DivPrune(CVPR 2025): avg accuracy at 64 tokens = 59.7 vs EADP = 62.2 (LLaVA-1.5-7B)
• On LLaVA-NeXT-7B at 640 tokens: CDPruner = 66.0 avg accuracy vs EADP = 66.3
• On Qwen2.5-VL at 256 tokens: CDPruner = around 72 avg accuracy vs EADP = 74.3
• On Qwen3-VL DocVQA at 256 tokens: DivPrune +7.0 points, CDPruner +9.8 points improvement by EADP
• At extreme pruning (32 tokens, 94.4%) EADP outperforms DivPrune by 2.5 points avg accuracy

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02484.

Fig 1: (a) illustrates a limitation of global guidance: it tends to attend to background

Fig 2: Overview of the EADP. EADP acts as a plug-and-play module compressing

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• The entropy threshold hyperparameter q for filtering text tokens requires tuning per dataset or task and may affect robustness.
• The approach requires computation of a full similarity matrix between text tokens and visual tokens, which may become expensive as N grows very large despite lightweight projection.
• The token-to-token similarity matrix for facility location selection can require O(N^2) memory if precomputed, limiting scalability on ultra-high resolution inputs.
• No explicit adversarial robustness evaluation is performed against attacks targeting token pruning or score manipulation.
• Evaluation is focused on pruning performance within existing VLM architectures; integration with end-to-end training of pruning modules was not explored.
• Limited analysis on latency improvements or hardware efficiency gains versus accuracy trade-offs beyond token counts.

Open questions / follow-ons

• How sensitive is the entropy-based token filtering method to different types of textual prompts or languages beyond English?
• Can the facility location submodular maximization be approximated with lower memory or complexity techniques for ultra-high resolution inputs?
• How would end-to-end trainable token pruning frameworks benefit from integrating entropy-aware denoising and submodular selection objectives?
• What are the latency and real-world deployment performance gains of EADP on edge devices or specialized hardware?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, EADP offers a principled method to compress large visual inputs while robustly preserving fine-grained visual cues in textual-visual queries. This has implications for CAPTCHA systems relying on multimodal AI models where inference costs and latency are critical yet fine-grained understanding is essential for security checks. The entropy-based filtering approach could be adapted to prune noise in textual inputs or cross-modal scores, improving robustness against ambiguous instructions that bots might exploit. Moreover, the submodular maximization formulation ensures compact but comprehensive visual representations, which can help prevent bypassing CAPTCHAs through partial image analysis. Overall, EADP’s disentangling of textual noise and structured token selection provides actionable insights that can be incorporated to improve multimodal model efficiency and fidelity in CAPTCHA and related bot-detection tasks.

Cite

@article{arxiv2607_02484,
  title={ Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning },
  author={ Xuehui Wang and Xuankun Yang and Wei Shen },
  journal={arXiv preprint arXiv:2607.02484},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02484}
}

Read the full paper

• arXiv:2607.02484 (abstract)
• arXiv:2607.02484 (PDF)