DnA: Denoising Attention for Visual Tasks

Source: arXiv:2606.27372 · Published 2026-06-25 · By Ron Campos, Subhajit Maity, Xin Li, Srijan Das, Aritra Dutta

TL;DR

This paper addresses a fundamental limitation in the standard softmax-based multihead attention (MHA) mechanism used in vision transformers (ViTs) for visual perception tasks. The authors observe that softmax attention suppresses potentially informative low or negative token interactions, resulting in noisy attention maps that dilute relevant features and reduce discriminability. They propose Denoising Attention (DnA), a novel attention mechanism that explicitly models positive and negative token interactions separately using two query sets and projects these into two distinct, quasi-orthogonal value subspaces. This subspace separation promotes better feature discrimination and suppresses irrelevant or distracting signals.

DnA replaces the traditional single softmax operation with a combination of softmax (positive query) and softmin (negative query) activations, weighted and combined with separate value projections. This design is theoretically motivated by information-theoretic principles and subspace classification error bounds linking larger principal angles between subspaces to improved classification accuracy. Empirically, DnA yields consistent improvements on ImageNet-1K (+0.8% top-1 accuracy vs. ViT-B baseline), demonstrates greater robustness on adversarial ImageNet-A, and transfers effectively to video tasks, improving action recognition by up to 4.0% on Toyota Smarthome and 1.2% on NTU RGB+D 60. Integrating DnA in video foundation models (Video LLMs) further enhances egocentric video question answering accuracy by 0.5% on the Ego-in-Exo PerceptionMCQ benchmark. Qualitative visualizations show sharper focus on relevant objects and suppression of distracting tokens compared to traditional and prior denoising attention baselines. The code and models are publicly released. Overall, DnA provides a theoretically sound, practically effective improvement for attention in visual transformers by leveraging both positive and negative interactions in separate subspaces.

Key findings

• DnA improves ImageNet-1K top-1 accuracy by an absolute 0.8% over ViT-B baseline (81.9% vs 81.1%).
• DnA surpasses alternative attention methods: cog attention and differential attention by approximately 0.5% accuracy on ImageNet-1K.
• On ImageNet-A adversarial robustness benchmark, DnA achieves higher accuracy (27.7% vs. 24.1%) and better calibration metrics (RMSCE 24.2 vs. 26.6, AURRA 40.7 vs. 33.5) than ViT-B.
• In video understanding with TimeSformer, replacing both space and temporal attention with DnA yields up to +4.0% mean class accuracy on Toyota Smarthome and +1.2% top-1 accuracy on NTU RGB+D 60.
• In ego-centric video question answering with Video LLMs (VideoLLaMA3), DnA improves average accuracy by 0.5% on all four PerceptionMCQ categories versus VisCoP baseline.
• Empirical subspace analyses show DnA produces larger principal angles between positive and negative value subspaces than differential attention, supporting theoretical error bound claims (Theorem 2 and 3).
• Qualitative attention visualizations demonstrate DnA attends more selectively to relevant objects and suppresses background noise compared to softmax and differential attention.
• DnA introduces only a moderate increase in parameters (100.7M vs. 86.6M) and computational demand (21.1 GFLOPs vs. 17.6) while maintaining equal throughput (909.1 img/s).

Threat model

The paper does not explicitly define a specific adversarial threat model. Instead, it frames the problem as one of improving representation quality under noisy or conflicting visual signals where irrelevant but closely associated distracting features confuse the attention mechanism. The implicit adversary can be viewed as distracting contextual objects that degrade classification or recognition performance. The model assumes a standard supervised setting without targeted adversarial attacks, manipulation, or spoofing attempts.

Methodology — deep read

The authors address the limitation in traditional softmax attention wherein softmax nonlinearly amplifies large positive token interactions but suppresses potentially informative negative or low interactions, resulting in noisy and less discriminative attentions.

Threat model and assumptions focus on standard visual classification and video understanding scenarios, without explicit adversarial attack settings. The adversary is implicitly represented by irrelevant or confusing visual features closely associated with correct class features.

Data provenance includes training from scratch on ImageNet-1K (1.28M images, 1000 classes) with images resized to 224x224. Additional benchmarks include ImageNet-A for robustness, CIFAR-10/100, Stanford Cars, MS COCO for downstream tasks; Toyota Smarthome and NTU RGB+D 60 for video understanding; and Ego-in-Exo PerceptionMCQ for egocentric video QA. Data splits and preprocessing follow standard protocols for each dataset.

Architecturally, DnA replaces the traditional single-query softmax attention in ViT-B. It uses two sets of queries (Q+ for positive interactions, Q− for negative interactions) interacting with the same keys. Each produces a scaled dot-product matrix normalized by softmax or softmin respectively. These attention weights separately weight two distinct learned value projections (V+ and V−) yielding two outputs. A learnable balancing parameter α_h weights the negative branch. These are then summed to produce the attention output for each head. This effectively models attention as a two-class classification problem, projecting onto two quasi-orthogonal subspaces with larger principal angles to reduce classification error (Theorem 2). Theoretical results (Theorem 3) and empirical analyses show subspace orthogonality is achievable and maintained.

Training uses the ViT-B backbone with 12 encoder layers, 12 heads per layer, patch size 16, and head dimension 64. Models are trained on NVIDIA H100 GPUs for 300 epochs following DeiT hyperparameters. For video tasks, pretrained weights from image training are leveraged and finetuned on video datasets for 15 epochs (video transformer) or 3 epochs (video LLM), with appropriate learning rates and schedules.

Evaluation protocols include top-1 and top-5 accuracy metrics on ImageNet-1K validation and test sets, robustness metrics RMS Calibration Error (RMSCE) and AURRA on ImageNet-A, mean class accuracy (mCA) on video action datasets, and accuracy on video QA benchmarks. Baselines include the ViT-B with traditional softmax attention, Cog Attention (which employs negative attention weights), and Differential Attention (which uses subtractive attention using shared value spaces). Multiple random seeds and repeated runs are reported for variance.

Visualization uses gradient-based attention interpretability extended to signed relevance for DnA and differential attention. Quantitative subspace analyses compute singular value decompositions of attention branches, measuring angles between principal vectors (intruder dimension analysis), across thousands of samples.

The paper releases code and pretrained weights for reproducibility. Some datasets like NTU and Video QA are public, but others such as EgoExo4D video Q&A training sets are proprietary or semi-public. Full hyperparameter details and ablations are included in supplementary material.

A concrete example end-to-end: On ImageNet-1K, image tokens are embedded and split into patches, then passed through 12 layers of ViT. At each self-attention layer, traditional softmax is replaced by dual-query DnA producing positive and negative attention weights; these weight separate value projections, combined to yield attention outputs with improved focus. The entire model is trained for 300 epochs with label supervision, eventually yielding a +0.8% accuracy gain versus baseline on the held-out test set.

Technical innovations

• Introducing dual queries (positive and negative) with separate softmax and softmin normalizations to explicitly capture positive and negative token interactions.
• Projecting positive and negative attention outputs onto two distinct learnable value subspaces with enforced quasi-orthogonality to promote subspace separation and improved class discriminability.
• Theoretical formulation linking principal angles between value subspaces to classification error bounds, motivating the two-subspace architecture.
• Adaptation of DnA to video transformers and video LLMs via replacing spatial, temporal, and cross-attention mechanisms, enabling denoising in spatiotemporal and multimodal contexts.

Datasets

• ImageNet-1K — 1.28 million images — public benchmark for image classification
• ImageNet-A — approximately 7,500 adversarially selected images — public adversarial robustness dataset
• CIFAR-10 and CIFAR-100 — 60,000 small images total — public
• Stanford Cars — 16,185 images — public fine-grained classification dataset
• MS COCO — 123,287 images with instance segmentation and detection labels — public
• Toyota Smarthome — 16,000 videos, 31 action classes — public video action recognition
• NTU RGB+D 60 — 57,000 videos, 60 action classes — public multi-view video dataset
• Ego-exo PerceptionMCQ — >40,000 QA pairs from egocentric videos — partially public/proprietary

Baselines vs proposed

• ViT-B baseline: ImageNet-1K top-1 accuracy = 81.1%, vs DnA = 81.9%
• Cog Attention: ImageNet-1K top-1 accuracy = 81.5%, vs DnA = 81.9%
• Differential Attention: ImageNet-1K top-1 accuracy = 81.5%, vs DnA = 81.9%
• ViT-B baseline ImageNet-A accuracy = 24.1%, vs DnA = 27.7%
• TimeSformer baseline on Toyota Smarthome (cross-subject) mean class accuracy = 67.5%, vs +DnA = 68.8%
• TimeSformer baseline on Toyota Smarthome (cross-view2) mean class accuracy = 59.5%, vs +DnA = 63.5%
• TimeSformer baseline on NTU RGB+D 60 (cross-subject) accuracy = 81.2%, vs +DnA = 82.4%
• VideoLLaMA3 baseline on Ego-in-Exo PerceptionMCQ average accuracy = 75.2%, vs +DnA = 75.7%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27372.

Fig 1: The original image from ImageNet-1K [18] shows the primary object, breast-

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 2: Original images (center) from ImageNet-1K, their attention activation maps

Fig 3: A schematic comparison between traditional attention and the proposed de-

Limitations

• The model increases parameters and computational cost compared to baseline ViT-B (about +16M parameters, +3.5 GFLOPs), although throughput is maintained; scalability to larger models or real-time scenarios is not studied.
• Robustness evaluation is limited to ImageNet-A adversarial samples but lacks extensive adversarial attack or distribution shift testing.
• The dual-query and dual-value projection increases model complexity; potential overfitting or training stability issues are not deeply analyzed.
• The video LLM adaptation uses a small subset of ego video QA data (∼46K pairs) which may limit generalization of findings.
• Theoretical results rely on assumptions like zero-mean Gaussian initialization and quasi-orthogonality which may not fully hold in practice.
• No ablation or experiments on tasks outside classification and video understanding, e.g. object detection or segmentation, are detailed in main results.

Open questions / follow-ons

• Can DnA's dual-query and dual-subspace approach be extended or adapted for large-scale language models or other modalities beyond vision?
• What is the performance and robustness of DnA under targeted adversarial attacks explicitly designed to fool attention mechanisms?
• Can the quasi-orthogonality constraints be enforced more explicitly or efficiently during training to ensure stable subspace separation?
• How would DnA perform on downstream tasks such as object detection, segmentation, or multimodal retrieval that require fine-grained localization?

Why it matters for bot defense

Bot-defense and CAPTCHA systems relying on visual recognition can suffer from noisy or ambiguous attention in detection and classification pipelines. The DnA mechanism offers a principled way to improve discriminative focus by simultaneously modeling both relevant and distracting features through separate attention branches and subspaces. This can help in rejecting or better classifying inputs that contain adversarial or confusing artifacts often exploited by bots or automated solvers. The improved calibration and robustness metrics on adversarial benchmarks like ImageNet-A also indicate potential for enhancing reliability in noisy environments typical in bot-defense scenarios. However, the added model complexity and parameter cost should be balanced against inference speed requirements common in CAPTCHA and bot filtering applications. Applying DnA into video and multimodal models also points towards improved capabilities in human activity and behavior analysis in security contexts. Neutral practitioners should consider this approach as part of the broader toolkit for enhancing attention robustness and discriminability in visual features, especially when subtle adversarial or contextual distractions occur.

Cite

@article{arxiv2606_27372,
  title={ DnA: Denoising Attention for Visual Tasks },
  author={ Ron Campos and Subhajit Maity and Xin Li and Srijan Das and Aritra Dutta },
  journal={arXiv preprint arXiv:2606.27372},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27372}
}

Read the full paper

• arXiv:2606.27372 (abstract)
• arXiv:2606.27372 (PDF)