Task-Specific Feature Fusion Method for Multi-Task Affective Behavior Analysis

Source: arXiv:2607.13986 · Published 2026-07-15 · By Jiajun Sun, Zhe Gao

TL;DR

This paper addresses the challenge of multi-task affective behavior analysis in-the-wild, specifically the ABAW11 Multi-Task Learning Challenge that requires simultaneous prediction of valence-arousal (VA), categorical expressions (EXPR), and facial action units (AU) from facial video frames. Unlike prior works that apply a single shared architecture for all tasks, the authors empirically show that these related but distinct tasks benefit from different visual features, temporal modeling approaches, feature fusion methods, and calibration techniques. They propose a task-adaptive feature fusion framework that leverages two pretrained and externally adapted visual backbones (DINOv2 ViT-L and DINOv3 ConvNeXt-base), frozen to produce complementary representations. Different downstream prediction heads, temporal processors, and fusion schemes are systematically explored and selected individually for each task based on validation performance. Their final system achieves a weighted overall validation score of 1.6341 on ABAW11, outperforming shared multitask learning (MTL) baselines by a substantial margin. This validates the hypothesis that task-specific fusion and processing strategies yield superior performance than imposing a unified MTL model for heterogeneous affective tasks.

Key findings

• The task-adaptive system achieves EXPR macro-F1=0.4222, AU macro-F1=0.5402, and mean VA CCC=0.6717 on the ABAW11 validation set, resulting in an overall score of 1.6341.
• For single-feature experts, ViT-L features outperform ConvNeXt-base for EXPR and AU, while VA strongly benefits from temporal modeling on both feature sets.
• Among fusion strategies, EXPR prefers feature concatenation with smoothing, AU benefits from late logit fusion with threshold calibration, and VA favors residual feature fusion with smoothing.
• Shared MTL baselines using the same frozen features reach at most 1.3598 overall validation score, significantly lower than task-adaptive fusion's 1.6341.
• In five-fold video-level cross-validation, a temporal convolutional network (TCN) expert for VA shows more stable performance (0.6411±0.0309 CCC) than residual fusion (0.5124±0.0378 CCC).
• LightGBM trained on frozen features is competitive for AU detection but underperforms for VA regression and EXPR classification.
• Different downstream architectures and calibration procedures are needed for optimal VA, EXPR, and AU performance due to their unique label types and affect representations.

Threat model

n/a - this work is focused on multi-task affective behavior recognition rather than security or adversarial threat analysis.

Methodology — deep read

1. Threat Model & Assumptions: The paper does not explicitly define a threat model since it focuses on multi-task affective analysis rather than adversarial robustness. It assumes a standard supervised learning setting centered on accurate affect prediction from video frames.

2. Data: The main data source is the official ABAW11 s-Aff-Wild2 dataset containing facial video frames with annotations for valence-arousal (continuous labels), expression categories (eight classes), and 12 facial action units with binary labels. Not all frames have valid labels for all tasks, so tasks are trained/evaluated on their valid subsets. To adapt the visual backbones, an external facial expression dataset was constructed from AffectNet and RAF-DB (eight expression classes). No Aff-Wild2 images were used during backbone adaptation. Frames are cropped, aligned, resized to 256x256, center-cropped to 224x224, and normalized with ImageNet statistics.

3. Architectures and Algorithms: They use two visual backbones adapted on the external expression set: (a) DINOv2 ViT-L/14 transformer with first 8 transformer blocks frozen, followed by adaptation with a mixture-of-experts MLP classifier, and (b) DINOv3 ConvNeXt-base finetuned with a linear expression classifier. Both produce 1024-dim features, which are frozen for downstream tasks on ABAW11.

Downstream, a pool of expert models is constructed including:

• Frame-level prediction heads: linear, MLP, and BigMLP with task-specific output sizes (2 for VA regression, 8-class logits for EXPR, 12 binary logits for AU). VA uses combined MSE and CCC loss; EXPR uses class-weighted cross-entropy with label smoothing; AU uses weighted binary cross-entropy plus per-AU threshold calibration.
• Temporal models: residual temporal convolutional networks (TCNs) with multiple dilated conv blocks on overlapping frame sequences (256 frames for EXPR/AU, 512 for VA) with GELU and dropout.
• Non-neural baseline: LightGBM models on ViT-L features augmented with local temporal mean and delta.
• Feature fusion strategies: direct concatenation, gated fusion (softmax gates), residual fusion (ViT-L main branch plus gated ConvNeXt feature), and late fusion at logit level with learned weights.
• Shared MTL models: joint training with shared MLP representation plus task-specific heads, a task-token transformer model with one attention layer, and residual task relation models.

4. Training Regime: Backbones are adapted on external data with standard augmentations, learning rate decay, and label smoothing. Downstream models are trained on ABAW11 training split; details on epochs and batch sizes are provided in appendices but not summarized in main text. LightGBM uses 300 trees, learning rate 0.03, max 31 leaves, subsampling and feature sampling at 0.9 and 0.8.

5. Evaluation Protocol: They evaluate on official ABAW11 validation set using EXPR macro-F1, AU macro-F1, and mean valence-arousal CCC metrics. The official overall score sums the CCC of valence and arousal plus the average macro-F1 for EXPR and AU. Ablations compare frame heads, single feature experts, fusion methods, temporal strategies, and MTL baselines. They perform 5-fold cross-validation ensuring no frames from the same video appear in both folds to assess stability.

6. Reproducibility: The paper details all architectural designs, loss functions, hyperparameters, and feature extraction steps. However, it does not mention public release of code or pretrained weights. The external expression dataset is constructed from public AffectNet and RAF-DB, but the exact subset is not released.

Concrete Example: For valence-arousal, the system extracts frozen 1024-d ViT-L and ConvNeXt features per frame, fuses them using a residual gate fusion method, applies a temporal convolutional network over 512-frame sequences, and produces frame-level VA predictions optimized using a combined MSE + CCC loss. Predictions are temporally smoothed and evaluated against ground truth continuous labels using CCC.

Overall, the work conducts an extensive empirical study of task-specific expert selection for MTL affective behavior analysis, carefully controlling variables by using frozen backbone features and comparing multiple downstream components in a unified experimental protocol.

Technical innovations

• Task-adaptive downstream architectures that select separate feature sources, fusion mechanisms, temporal models, and calibration methods for each affective task (VA, EXPR, AU) rather than a shared MTL model.
• Combination of two complementary frozen, pretrained visual backbones (DINOv2 ViT-L transformer and DINOv3 ConvNeXt-base convolutional) adapted on external facial expression data to extract features providing complementary global and local representations.
• Systematic comparison of frame-level vs temporal convolutional heads, neural vs tree-based models (LightGBM), multiple fusion strategies (concatenation, gated fusion, residual fusion, late logit fusion), and threshold calibration for multi-label AU detection.
• Demonstration that frozen backbone features combined with task-specific post-hoc temporal smoothing and fusion suffice to achieve state-of-the-art performance on ABAW11 challenge without end-to-end backbone finetuning.

Datasets

• s-Aff-Wild2 (ABAW11 official dataset) — size unspecified in paper — official benchmark for affective behavior analysis in-the-wild
• External expression set — constructed from AffectNet and RAF-DB — used for backbone adaptation; no Aff-Wild2 images included
• AffectNet — large-scale facial expression and valence/arousal dataset
• RAF-DB — real-world facial expression database

Baselines vs proposed

• Shared MTL Task-token model: overall validation score = 1.3598 vs proposed task-adaptive fusion: 1.6341
• ViT-L single-feature EXPR macro-F1 = 0.3651 vs task-adaptive fusion for EXPR = 0.4222
• ViT-L single-feature AU macro-F1 = 0.5252 vs task-adaptive fusion for AU = 0.5402
• ViT-L single-feature VA CCC = 0.6367 vs task-adaptive fusion for VA = 0.6717
• LightGBM on ViT-L features for AU macro-F1 = 0.5162 vs neural fusion AU macro-F1 = 0.5402
• Five-fold cross-validation for VA: temporal convolutional network CCC = 0.6411±0.0309 vs residual fusion CCC = 0.5124±0.0378

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.13986.

Fig 1: presents the overall workflow of our system. The pipeline consists of

Limitations

• Backbones are frozen after external adaptation; no end-to-end fine-tuning on target ABAW11 multi-task dataset which could further improve performance.
• Task-adaptive expert selection relies on validation-based grid search rather than learned or dynamic task weighting mechanisms, limiting scalability.
• Official test set results are not reported as the ABAW11 challenge final leaderboard was not yet available at writing; conclusions rely solely on validation splits.
• The external facial expression set used for backbone adaptation is not publicly released exactly, potentially hindering reproducibility.
• No adversarial or robustness evaluations against occlusion, pose variation, or cross-domain shifts beyond the wild ABAW11 data.
• Temporal models only use convolutional networks without exploring advanced temporal architectures like transformers or attention mechanisms.
• The LightGBM baseline underperforms on regression tasks, suggesting limited suitability to continuous affect estimation.

Open questions / follow-ons

• Can end-to-end multi-task training with backbone finetuning improve performance beyond the frozen-feature regime?
• How would automated task-adaptive expert weighting or mixture-of-experts learning perform versus manual validation selection?
• What robustness gains can be achieved by incorporating advanced temporal attention or transformer models for affective temporal dynamics?
• How transferable are these task-adaptive fusion strategies to other multi-task emotion recognition datasets or modalities (e.g., audio-visual)?

Why it matters for bot defense

Bot-defense and CAPTCHA systems requiring facial affect analysis or behavioral biometrics can draw from this paper's insight that related but distinct affect recognition tasks benefit from task-specific architectures and feature fusions rather than a monolithic shared model. For example, systems detecting emotional states, expressions, or subtle facial actions to differentiate humans from bots might improve performance by applying tailored fusion strategies for each prediction output. The systematic evaluation of pretrained frozen features combined with post-hoc temporal smoothing suggests a resource-efficient approach that avoids expensive end-to-end retraining while still providing competitive accuracy. Furthermore, the empirical observation that temporal modeling benefits continuous regression tasks like valence-arousal prediction may inform design choices for dynamic human-bot interaction scenarios where time-evolving affect cues are relevant. However, the lack of adversarial or cross-domain robustness studies indicates further validation is needed before deployment in security-critical CAPTCHA or bot-defense applications.

Cite

@article{arxiv2607_13986,
  title={ Task-Specific Feature Fusion Method for Multi-Task Affective Behavior Analysis },
  author={ Jiajun Sun and Zhe Gao },
  journal={arXiv preprint arXiv:2607.13986},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.13986}
}

Read the full paper

• arXiv:2607.13986 (abstract)
• arXiv:2607.13986 (PDF)