Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

Source: arXiv:2606.30611 · Published 2026-06-29 · By Seongro Yoon, Donghyeon Cho, Jinsun Park, François Brémond

TL;DR

This paper addresses a key limitation in Vision Transformer (ViT) based video models for facial expression recognition (FER): the tendency of standard self-attention mechanisms to focus on dominant global motions and coarse temporal dynamics, which diminishes sensitivity to subtle and localized facial expression cues. The authors propose MiRA (Marginal-induced Attention Redistribution), a plug-in module that reweights framewise attention in ViTs to enhance spatio-temporal selectivity toward fine-grained facial dynamics without adding trainable parameters. MiRA derives frame-level confidence and intra-frame concentration statistics from self-attention maps to estimate frame marginal importance, redistributing attention away from noisy or redundant frames toward subtle facial movements.

MiRA is presented in two modes: an exact post-softmax redistribution mode that explicitly rescales attention maps, and a more efficient pre-softmax approximation called flashLite mode that integrates with FlashAttention kernels for reduced memory input/output while retaining accuracy. Experiments on challenging FER benchmarks (DFEW, MAFW, FERV39k) show consistent improvements over strong ViT baselines pretrained on VoxCeleb2 in an unsupervised mask-video-prediction paradigm. Ablations confirm the benefits of reweighting across all transformer layers and combining confidence with intra-frame concentration. Compared to baseline framewise or augmented attention variants, MiRA better captures informative, temporally sparse, and spatially localized facial expression cues. FlashLite achieves nearly the same accuracy as exact mode while reducing latency by 20%. MiRA outperforms recent general video models and achieves new state-of-the-art results among video-only FER approaches, competitive with more complex multi-modal methods.

Key findings

• Applying MiRA reweighting across all 12 transformer layers yields best performance: DFEW UAR 63.28% & WAR 75.57%, compared to 59.94%/72.90% for vanilla VideoMAE baseline (Tab. 1, Tab. 4).
• Balanced weighting of frame-level confidence and intra-frame concentration scores (wcon=0.5, went=0.5) achieves near-optimal results on DFEW and MAFW (Tab. 2).
• Power factor β=1.5 for framewise prior sharpening consistently yields best results in both shallow (κ=2) and full-depth (κ=12) reweighting (Tab. 3).
• MiRA outperforms naive framewise modeling baselines: reused intra-frame attention, framewise augmented blocks, and parameter-matched variants, indicating the effectiveness of marginal-induced redistribution (Tab. 4).
• FlashLite mode approximates exact mode redistribution closely: DFEW UAR 63.77% vs 63.88%; MAFW UAR 39.20% vs 39.67% averaged over 5 folds, with a 20% latency reduction and 26% throughput gain on an NVIDIA H100 GPU (Tab. 5, Fig. 4).
• MiRA achieves state-of-the-art video-only FER accuracy on DFEW (UAR 65.29%) and remains competitive on MAFW and FERV39k, surpassing prior self-supervised ViT-based models like VideoMAE and AdaMAE (Tab. 6, Tab. 7).
• Redistributed attention maps preserve query–key structure, introducing only mild perturbations while enhancing frame-level temporal selectivity, as shown by low JS divergence and entropy difference metrics (Fig. 4).

Threat model

n/a - This work is not a security paper and does not define an adversarial threat model. The focus is on improving transformer attention mechanisms to better capture subtle facial expression dynamics in uncontrolled video, assuming no malicious adversary but rather aiming to suppress noisy or dominant global motions that obscure relevant expression cues.

Methodology — deep read

1. Threat Model & Assumptions: The paper focuses on improving attention attribution in Vision Transformer backbones for video-based facial expression recognition under uncontrolled, in-the-wild conditions, where frames may contain dominant global motions (e.g., head pose changes) and temporal redundancy. The threat is not adversarial but revolves around suppressing noisy or irrelevant attention allocations to better highlight subtle facial expression cues localized spatially and temporally. The model assumes access to pretrained ViT encoder backbones and large-scale unlabeled or weakly supervised videos for downstream fine-tuning.

2. Data: The backbone is pretrained using VideoMAE on VoxCeleb2, a large corpus of over 1.2 million real-world face video utterances with natural affective variations. Fine-tuning and evaluation utilize three FER benchmark datasets: DFEW (~12K clips, 7 emotions), MAFW (~10K clips, 11 emotions), and FERV39k (~39K clips, 7 emotions), all in unconstrained video conditions. Standard 5-fold cross-validation is used for evaluation on DFEW and MAFW. No explicit face cropping or alignment is required.

3. Architecture/Algorithm: MiRA is a plug-in attention reweighting module integrated into each self-attention block of the ViT backbone. It operates on the self-attention maps after softmax (exact mode) or on attention logits before softmax (flashLite). MiRA computes frame-level confidence scores c_t by summing attention weights assigned to token groups corresponding to each video frame (key side). It also computes intra-frame concentration H_t as inverse entropy over spatial token attention distributions within a frame, capturing how localized the attention is within that frame.

These two statistics are combined linearly (weights w_con and w_ent) to produce a frame-marginal importance score f_t. This score is min-max normalized and sharpened via power-softmax (exact mode) or temperature-controlled softmax (flashLite). Target frame priors π_t are derived from these normalized scores.

Attention from the original transformer is reweighted by computing multiplicative scaling factors α_t = clip(π_t / c_t, α_min, α_max), which adjust attention mass per frame to align marginals with π_t while preserving original query-key distributions via ratio normalization and residual interpolation. FlashLite avoids explicit post-softmax attention computation by approximating attention statistics using average squared L2 norms of key vectors as proxies. Scaling factors α_t are injected as additive log-biases directly into the attention logits before softmax, enabling efficient streaming computation compatible with FlashAttention's fused kernels.

4. Training Regime: The VideoMAE ViT-B backbone is pretrained on VoxCeleb2 with masked video modeling for self-supervised representation learning. For fine-tuning, the MiRA module is enabled with fixed hyperparameters: w_con = 0.5, w_ent = 0.5, β=1.5, τ=1.7. Fine-tuning is performed at various reweighting depths κ (number of top layers with MiRA) to assess impact. The hardware includes NVIDIA H100 GPUs, needed for profiling latency and throughput (flashLite mode). No additional trainable parameters are introduced by MiRA, avoiding extra training complexity.

5. Evaluation Protocol: Evaluations are performed on DFEW, MAFW, and FERV39k using Unweighted Accuracy Rate (UAR) and Weighted Accuracy Rate (WAR). Protocols follow standard 5-fold cross-validation splits where applicable. Baselines include vanilla VideoMAE ViT-B, framewise augmented attention baselines, and recent state-of-the-art video FER models, both self-supervised and supervised. Ablations study hyperparameters and applying MiRA at different depths. Behavioral analyses measure changes in attention maps using Frobenius distance, Jensen-Shannon divergence of framewise attention, entropy differences, and cosine similarity. Latency and throughput are measured on hardware.

6. Reproducibility: Code and pretrained models are publicly available at https://github.com/ysrinria/MiRA. Datasets like VoxCeleb2, DFEW, MAFW, and FERV39k are publicly accessible following their own licenses. Hyperparameters and architectural details are documented in appendices. Some experiments rely on FlashAttention optimized kernels. Overall, the methodology is clearly described to reproduce training and inference.

Example End-to-End: Given a pretrained VideoMAE ViT-B on VoxCeleb2, MiRA is integrated into all 12 self-attention blocks during fine-tuning on DFEW. For each attention block forward pass, frame-level confidence and intra-frame concentration scores are computed from post-softmax attention maps. These statistics produce redistributive scaling factors α_t applied to reweight attention toward informative, subtle facial expression frames. This attention redistribution sharpens spatio-temporal focus, improving recognition accuracy from 59.94% UAR baseline to 63.28% UAR using full-depth MiRA. FlashLite mode approximates the same effect efficiently by modulating attention logits using a key-energy surrogate, yielding nearly equal accuracy with 20% faster inference.

Technical innovations

• Introduction of frame-marginal attention redistribution (MiRA) that combines frame-level confidence and intra-frame spatial concentration scores to selectively reweight ViT self-attention toward subtle facial expression dynamics.
• Formulation of an exact post-softmax redistribution mode that rescales attention marginals per frame using ratio-based alignment and residual interpolation preserving query-key structure.
• Development of flashLite mode, a pre-softmax approximation integrating framewise modulation biases into FlashAttention kernels to enable efficient streaming attention computation with reduced memory I/O.
• Use of average key vector squared norm as a proxy for frame-level importance in flashLite mode, eliminating the need to materialize full attention maps and enabling runtime-efficient attention reweighting.

Datasets

• VoxCeleb2 — 1.2M+ video utterances — public speaker recognition corpus used for self-supervised pretraining
• DFEW — ~12K video clips, 7 emotion labels — public facial expression recognition benchmark
• MAFW — ~10K clips, 11 emotions — public FER benchmark from movies and TV shows
• FERV39k — ~39K clips, 7 expression categories — large-scale in-the-wild FER dataset from online videos

Baselines vs proposed

• Vanilla VideoMAE ViT-B: DFEW UAR = 59.94%, WAR = 72.90% vs MiRA Exact (κ=12): UAR = 63.28%, WAR = 75.57% (Tab. 4)
• Baseline framewise attention variants (reused intra-frame, framewise augmented block + param-matched): DFEW UAR ≤ 62.68% vs MiRA Exact 63.28% (Tab. 4)
• VideoMAE: DFEW UAR = 59.94%, WAR = 72.90%; MiRA Exact: UAR = 63.88%, WAR = 76.21% (Tab. 6)
• FlashLite vs Exact mode (5-fold average on DFEW): UAR 63.77% vs 63.88%, WAR 76.51% vs 76.21%; latency reduced 20% (Tab. 5)
• VideoMAEv2, AdaMAE, MGMAE: DFEW UAR ~59–61% vs MiRA Exact 63.88% (Tab. 6)
• MiRA ViT-B surpasses MAE-DFER and SVFAP video-only methods in UAR/WAR on DFEW and MAFW (Tab. 7)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.30611.

Fig 1: Frame-marginal attention reweighting modules. (b) Exact mode applies

Fig 2: Illustration of the effects of frame-level confidence and intra-frame

Fig 3: Illustration of baseline attention variants.

Fig 4: Behavioral analysis of Exact and FlashLite modes. (a,b) Attention per-

Fig 5: Global and local attention visualizations during masked video recon-

Fig 6 (page 15).

Fig 6: Trans-

Fig 7: Representative examples from SAMM (left) and MMEW (right).

Limitations

• MiRA assumes the ViT backbone already pretrained with masked video modeling; gains unexplored for other architectures or plain supervised training.
• FlashLite’s approximation relies on key-energy proxies which provide weaker framewise contrast than exact mode, potentially limiting effectiveness in some scenarios.
• Evaluation is limited to FER datasets; applicability to other fine-grained dynamic video tasks unclear.
• No explicit adversarial robustness or out-of-distribution tests conducted to assess defense against corrupted or adversarial facial videos.
• Frame-level priors are hand-tuned with fixed hyperparameters without adaptive learning; potential gains from data-driven weighting unexplored.
• MiRA does not introduce additional trainable parameters but adds computation and memory overhead in exact mode; flashLite addresses this but detailed cost-benefit tradeoffs under resource constraints need further quantification.

Open questions / follow-ons

• How well does MiRA generalize to other video understanding tasks requiring fine-grained spatio-temporal attention beyond facial expression recognition?
• Can the frame-level weighting parameters be learned adaptively or incorporated into end-to-end training to further improve reweighting effectiveness?
• What is the impact of MiRA on robustness to adversarial perturbations or common video corruptions affecting facial regions?
• How does MiRA interact with multimodal fusion approaches (audio, language) for facial behavior understanding?

Why it matters for bot defense

For practitioners designing bot-defense or CAPTCHA systems that incorporate facial expression or behavior analysis, this work highlights a novel mechanism for enhancing transformer-based video models' sensitivity to subtle, localized facial motions often obscured by head or camera movements. MiRA provides a plug-in approach to shift attention toward informative temporal frames without retraining the entire model or increasing parameters, which could improve signal quality in dynamic facial cues relevant for liveness detection or behavioral biometrics. The flashLite mode's compatibility with efficient attention kernels also makes real-time deployment more feasible.

At a conceptual level, the paper’s frame-marginal attention redistribution mechanism exemplifies how incorporating domain-relevant locality and saliency priors into generic video transformer models can yield more meaningful spatio-temporal representations. This approach may inspire analogous methods in CAPTCHA challenges that rely on fine-grained behavioral or skeletal motion cues, offering improved robustness against bots by focusing on critical temporal moments rather than global or spurious motions. However, the paper does not address adversarial robustness or explicit bot attack scenarios, so additional security validation would be necessary before adoption in adversarial settings.

Cite

@article{arxiv2606_30611,
  title={ Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding },
  author={ Seongro Yoon and Donghyeon Cho and Jinsun Park and François Brémond },
  journal={arXiv preprint arXiv:2606.30611},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30611}
}

Read the full paper

• arXiv:2606.30611 (abstract)
• arXiv:2606.30611 (PDF)