CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition

Source: arXiv:2607.13976 · Published 2026-07-15 · By Tung Hung Bui, Hong Hai Nguyen, Van Thong Huynh

TL;DR

The paper addresses the challenge of detecting ambivalence and hesitancy (AH) in unconstrained video, a task complicated by inherent ambiguity and subtle cross-modal incongruence rather than prototypical affective signals. The authors present CF-Net, a multimodal network combining frozen large pretrained backbones for visual (SigLIP2), audio (HuBERT), and textual (DistilBERT) streams. CF-Net applies per-speaker normalization to reduce identity leakage, and fuses modalities through a ConflictFusion module that explicitly encodes pairwise cross-modal differences to capture incongruence. Training combines certainty-weighted focal loss, manifold mixup, modality dropout, and an auxiliary certainty regression head to leverage annotation uncertainty and improve robustness on ambiguous cases. This approach balances the need to model cross-modal disagreement, reduce speaker identity overfitting, and handle label uncertainty. CF-Net achieves competitive Macro F1 scores of 0.7155 on the BAH validation set and 0.7364 on a private ABAW test set, with positive validation-to-test transfer showing improved generalization over prior end-to-end fine-tuned systems.

Key findings

• Per-speaker normalization of visual and audio backbone features improves validation Macro F1 by +0.013 and yields a +0.021 gain on test performance, indicating reduced speaker-identity overfitting.
• Use of an auxiliary certainty-regression head combined with certainty-weighted focal loss adds +0.013 incremental gain in validation Macro F1 by effectively weighting ambiguous samples.
• CF-Net achieves Macro F1 = 0.7155 on the BAH validation split (124 clips) and 0.7364 (AP = 0.7392) on the 11th ABAW private test set (152 clips), surpassing prior frozen-backbone models.
• ConflictFusion multimodal fusion outperforms simple mean fusion by 1.8 percentage points in Macro F1, demonstrating the value of explicitly encoding pairwise cross-modal absolute differences.
• Training techniques including manifold mixup (α=0.2) and modality dropout (p=0.15) cumulatively contribute +3.7 points to validation Macro F1, reducing overfitting on the small 778-video training set.
• Using BiGRU temporal encoders instead of BiLSTM improves validation Macro F1 from 0.7110 to 0.7136, indicating modest gains from architecture choice.
• Frozen large pretrained backbones with light temporal encoding allow training to converge in under 15 minutes on a single RTX 4060 Ti GPU, enabling efficient experimentation.
• Validation-to-test performance gap is reversed (+0.021 improvement) compared to prior fine-tuned systems that suffer overfitting, confirming robustness to unseen speakers.

Threat model

The adversary is abstracted as the domain shift caused by unseen speakers in validation and test sets, who have never been observed during training. The system must generalize without access to any adaptation or enrollment data for these speakers. The model cannot rely on memorizing speaker identity cues, and must handle label uncertainty due to genuine perceptual ambiguity in the annotations.

Methodology — deep read

1. Threat model and assumptions: The adversary is implicitly the data split enforcing strict speaker disjointness between training, validation, and test sets, requiring generalization to unseen speakers with no access to adaptation data. The system must avoid overfitting speaker identity cues and handle label uncertainty stemming from genuinely ambiguous ambivalence/hesitancy samples.

2. Data: The study uses the BAH dataset from the ABAW 3rd AH challenge, comprising 1427 video clips from 300 unique speakers, partitioned into 778 training, 124 validation, and 525 test videos with no speaker overlap. Each clip has a binary ambivalence/hesitancy label plus an annotator-agreement certainty score scaled from 1 (low) to 3 (high). The label distribution is roughly balanced, with ~49.5% AH in training and 60.5% in validation/test sets.

3. Architecture/algorithm: Visual features are extracted from frames (1 every 5 frames at 30fps) using frozen SigLIP2 base model producing 768-dimensional embeddings. Audio is resampled to 16 kHz, chunked with overlap, and encoded via frozen HuBERT base to 768-d embeddings aggregated per visual frame. Text features come from manual transcripts tokenized and encoded by frozen DistilBERT, mean pooled to a single 768-d vector. Visual/audio features undergo per-speaker normalization by subtracting each speaker's mean embedding to remove identity bias.

Normalized visual/audio sequences are passed through independent bidirectional GRUs (single layer, hidden 128 per direction, output projected to 128-d) with MIL attention pooling. Text is projected via a linear-ReLU-dropout encoder to 128-d. The ConflictFusion module concatenates the three modality embeddings ev, ea, et and all pairwise elementwise absolute differences |ev–ea|, |ev–et|, |ea–et| into a 768-d fused vector.

This vector feeds two heads: a binary AH classifier with two linear layers and ReLU + dropout producing sigmoid output, and an auxiliary certainty regression head predicting the annotator certainty used only during training to regularize ambiguous samples.

4. Training regime: Only the temporal encoders, pooling, and heads are trained; all backbones are frozen. Adam optimizer with learning rate 3e-4 and weight decay 1e-4, batch size of 8, early stopping with patience 8 epochs on validation Macro F1, maximum 30 epochs. Training on a single NVIDIA RTX 4060 Ti (~15 min/runtime). Fixed random seed 1234 for reproducibility.

Augmentations include manifold mixup on fused features with Beta(0.2,0.2) λ, modality dropout (15% dropout independently on each modality), and a certainty-weighted focal loss that scales sample gradient contribution proportional to the annotator certainty score, encouraging learning from reliable labels while reducing overconfidence on ambiguous examples.

5. Evaluation protocol: Metrics are Macro F1 and Average Precision on positive class. Validation is the 124 video split; final testing is done on a private 152-video set held out by the challenge server. Ablations isolate component contributions by repeated runs on validation split.

6. Reproducibility: Backbone weights and feature extraction are frozen and precomputed; code release is not specified but experiments are fully described. Dataset is public but private test set held by challenge organizers.

Example end-to-end: A video clip is preprocessed—frames extracted and face cropped, audio chunked, transcript tokenized. Features are extracted via frozen SigLIP2, HuBERT, DistilBERT. Visual/audio embeddings are normalized per speaker by subtracting the speaker's mean vectors computed from all clips. Temporal encoders and MIL attention pool each stream into 128-d vectors. The ConflictFusion module forms a 768-d fused vector combining embeddings and their pairwise absolute differences. This vector feeds the AH classifier producing a probability and a certainty regressor during training. Loss is computed using a focal loss weighted by the certainty, plus MSE on certainty prediction, guiding parameter updates on the GRU, pooling, and heads only. This pipeline achieves robust prediction on ambiguous and visually/auditorily confounding clips across unseen speakers.

Technical innovations

• The ConflictFusion module explicitly appends pairwise elementwise absolute differences between modality embeddings to capture cross-modal incongruence critical for ambivalence/hesitancy.
• Per-speaker feature normalization applied to frozen visual and audio embeddings removes speaker identity bias without requiring subject-specific adaptation data.
• Certainty-weighted focal loss combined with an auxiliary certainty-regression head jointly leverage annotator agreement scores to modulate training focus and reduce overconfidence on ambiguous samples.
• Training regularization through manifold mixup and modality dropout improves robustness against small training set overfitting while preserving sensitivity to subtle multimodal signals.

Datasets

• BAH dataset — 1,427 videos from 300 speakers — publicly released via ABAW challenge

Baselines vs proposed

• Organiser baseline: test Macro F1 = 0.2827 vs CF-Net test Macro F1 = 0.7364
• Team LEYA (fine-tuned large models): validation F1 = 0.830 vs CF-Net val F1 = 0.716
• Team LEYA: test F1 = 0.714 vs CF-Net test F1 = 0.7364
• Team Fennec (ConflictAwareAH): val F1 = 0.746 vs CF-Net val F1 = 0.716
• Team BROTHER ensemble: val F1 = 0.736 vs CF-Net val F1 = 0.716
• CF-Net vs mean fusion multimodal baseline: +1.8 pp Macro F1 on validation
• Auxiliary certainty head + certainty-weighted focal loss: +0.013 pp Macro F1
• Manifold mixup + modality dropout: +3.7 pp Macro F1 cumulative gain

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.13976.

Fig 2: Predicted probability distributions P(AH=1) of CF-Net on the

Limitations

• The backbone encoders remain frozen; results might improve with fine-tuning though risk overfitting.
• The ConflictFusion fusion is fixed concatenation of differences rather than a learnable attention mechanism which may limit flexible weighting of modality conflicts.
• No explicit adversarial evaluation or out-of-domain tests beyond speaker disjointness.
• The BAH dataset is small with only 778 training samples and 124 validation, limiting generalization and making overfitting a constant risk.
• Unclear if proposed methods generalize beyond scripted clinical motivational interviewing contexts to naturalistic or other social settings.
• Code and pretrained fusion models are not publicly released to verify exact implementation details.

Open questions / follow-ons

• Would selective fine-tuning of the top layers of SigLIP2 or HuBERT under stronger regularization improve modality feature quality without overfitting speaker identity?
• Can replacing ConflictFusion's fixed concatenation with a learned cross-modal attention mechanism enable better weighting of conflicting modalities per sample?
• Could domain-adversarial training or other domain adaptation approaches further improve robustness to zero-speaker-overlap constraints?
• How generalizable is CF-Net to other domains featuring ambivalence/hesitancy outside clinical motivational interviews?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, CF-Net's approach offers a valuable perspective on detecting subtle, ambiguous signals manifested through cross-modal incongruence rather than clear-cut affective cues. The explicit ConflictFusion fusion that encodes modality disagreement could inspire CAPTCHA challenge designs that test coherence across user input modalities—for example, voice and text consistency or facial cues mismatches signaling hesitancy or deception. The lightweight frozen-backbone architecture and speaker normalization techniques demonstrate practical methods to prevent overfitting on identity cues, an important consideration in user verification systems to avoid spoofing or replay attacks based on learned biometric data. Certainty-weighted training accounting for annotation ambiguity suggests that system confidence calibration and handling borderline cases with reduced certainty can improve robustness, an insight useful in bot detection to reduce false positives on ambiguous behavior. While CF-Net targets ambivalence/hesitancy in video, its architectural principles and training strategies offer transferable concepts for designing multimodal, uncertainty-aware systems that need to generalize well to unseen users and subtle adversarial patterns.

Cite

@article{arxiv2607_13976,
  title={ CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition },
  author={ Tung Hung Bui and Hong Hai Nguyen and Van Thong Huynh },
  journal={arXiv preprint arXiv:2607.13976},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.13976}
}

Read the full paper

• arXiv:2607.13976 (abstract)
• arXiv:2607.13976 (PDF)