SpEmoC: A Balanced Speaker-Segment Multimodal Emotion Benchmark

Source: arXiv:2607.18109 · Published 2026-07-20 · By Sania Bano, Shahzad Ahmad, Santosh Kumar Vipparthi, Sukalpa Chanda, Subrahmanyam Murala

TL;DR

SpEmoC addresses major limitations in multimodal emotion recognition datasets by providing a large-scale, balanced, and temporally aligned benchmark focused on speaker segments in conversational contexts. The authors curate 30,000 high-quality emotion-labeled clips from an initial pool of over 306,000 extracted speaking segments sourced from 3,100 English-language movies and TV series. A hybrid annotation pipeline fuses pretrained text and audio emotion models with human expert validation to ensure accuracy and class balance across seven Ekman emotion categories, including minority classes like Fear and Disgust. Importantly, SpEmoC uses strict movie- and series-level splits to avoid content leakage, enabling robust evaluation of generalization and transfer.

Extensive benchmarking with state-of-the-art multimodal models (MulT, MISA, TCL-MAP, EMOE, EmotionCLIP) shows substantial improvements over prior datasets MELD and CAER in balanced per-class F1 scores and reduced bias towards dominant Neutral emotions. Cross-dataset transfer experiments demonstrate that models trained on SpEmoC generalize more stably and maintain performance on minority classes. Additional analyses including low-data regimes and unimodal versus multimodal inputs provide insights on SpEmoC's utility for robust, transferable affect recognition. Overall, SpEmoC advances empirical understanding of dataset design impacts on multimodal emotion recognition and provides an open resource for benchmarking.

Key findings

• SpEmoC contains 30,000 balanced clips curated from 306,544 raw speaking segments across 3,100 English movies/TV series.
• Seven emotion classes balanced in SpEmoC: Anger, Disgust, Fear, Joy, Sadness, Surprise, Neutral, with minority classes (Fear, Disgust) well represented.
• Movie- and series-level splits prevent content overlap in train/val/test sets, ensuring unseen test scenarios.
• Models trained on SpEmoC achieve 30+ absolute Macro-F1 point improvements over MELD and CAER, e.g., EMOE Macro-F1: 70.25 (SpEmoC) vs 38.19 (MELD) and 15.54 (CAER).
• Imbalance gap (Weighted F1 - Macro F1) reduced near zero on SpEmoC (∆ ≤ 0.65) vs 15-25 on MELD/CAER, indicating improved balanced recognition.
• Cross-dataset transfer: SpEmoC-trained models maintain minority-class F1 scores on Fear and Disgust > 30, while MELD/CAER-trained collapse near zero under domain shift.
• Filtering pipeline removed ~85% neutral clips using multimodal confidence thresholds, preserving a small fraction to maintain distributional realism.
• Inter-annotator agreement on final labels is substantial (Fleiss’ κ = 0.62), based on 3+ experts per clip using visual, audio, text.

Threat model

n/a — The paper does not propose a security model or adversarial threat setting. Its focus is to build a robust, balanced multimodal emotion recognition benchmark to improve generalization and minority class performance.

Methodology — deep read

Threat Model & Assumptions: The paper focuses on building a robust multimodal emotion recognition benchmark rather than security adversaries. The main challenge is to control dataset biases that cause poor generalization, especially on minority emotions. Assumptions include availability of naturalistic movie/TV dialogues and that models have access to text, audio, and visual modalities.

Data Collection: 3,100 English-language movies/series with durations ≥40 minutes were collected from YouTube. Whisper ASR transcriptions with word-level timestamps were used for dialogue segmentation. Utterances of ≥12 words ending with terminal punctuation formed initial segments, totaling ~306,544 clips.

Multimodal Alignment: For each segment, synchronized video frames (~30 million total), audio waveforms, and text transcripts were extracted using timestamps. Face and human detections ensured presence of visible speakers (YOLOv8 with ≥90% frame presence). Alignment correctness was enforced with <0.1 s audio-video duration mismatch tolerance.

Annotation Pipeline: A hybrid multimodal annotation scheme combined pretrained DistilRoBERTa for text emotion logits and Wav2Vec 2.0 for audio emotion logits, both mapping to Ekman’s seven emotions. A logit-based fusion mechanism with KL divergence regularization encouraged agreement between text and audio predictions and produced soft pseudo-labels. Thresholds on neutral-class scores and face presence filtered out weak emotional or low-quality segments, reducing to ~50,000 candidate clips.

Human Validation: 20 expert annotators independently reviewed filtered clips across modalities, applying majority voting to finalize labels. Fleiss' kappa was 0.62, indicating substantial agreement. Ambiguous clips were discarded, resulting in 30,000 balanced emotion clips.

Dataset Splitting: Strict movie- and series-level splits were used assigning entire franchises exclusively to train (70%), val (10%), or test (20%) sets to prohibit content overlap and simulate real-world generalization.

Benchmarking Setup: Five state-of-the-art multimodal models (MulT, MISA, TCL-MAP, EMOE, EmotionCLIP) were trained and evaluated with consistent protocols on SpEmoC, MELD, and CAER. Metrics included Weighted F1, Macro F1, per-class F1, and imbalance gap (difference). Cross-dataset evaluations tested transfer from each dataset to the others without adaptation. Additional ablations included low-data training, neutral-class removal, and unimodal vs multimodal inputs.

Example End-to-End: For a given movie clip, the ASR segment is extracted based on linguistic cues, then aligned audio and video frames are retrieved. DistilRoBERTa and Wav2Vec models produce emotion logits, which are fused with KL divergence penalty. Clips with low neutral logits and high face presence pass filtering. Human annotators review these with audiovisual context, produce final emotion labels based on majority vote, after which the clip is added to the curated balanced dataset for training and evaluation.

Reproducibility: The authors plan public release under an EULA for non-commercial research with extensive documentation. Supplemental materials detail splits, annotation files, and preprocessing steps. Code or pretrained model weights availability is not explicitly stated.

Technical innovations

• A scalable hybrid multimodal annotation pipeline combining pretrained text and audio emotion models with logit-level fusion and KL divergence regularization to yield high-quality pseudo-labels for large unlabeled datasets.
• A large-scale dataset curation strategy using stringent face-presence verification and neutral-class logit thresholding to achieve a near-balanced emotional distribution across seven categories.
• Strict movie- and series-level data splits that prevent overlap of any actors, scenes, or dialogues between training, validation, and test sets to ensure robust generalization evaluation.
• Comprehensive benchmarking across multiple SOTA multimodal emotion recognition architectures under identical training protocols to isolate dataset design effects on model performance and robustness.

Datasets

• SpEmoC — 306,544 raw clips, 30,000 curated clips — English movies and TV series from YouTube
• MELD — ~13,000 clips — TV series Friends
• CAER — ~13,000 clips — 79 TV shows

Baselines vs proposed

• EMOE on MELD: Macro-F1 = 38.19; on CAER: 15.54; on SpEmoC: 70.25
• TCL-MAP on MELD: Macro-F1 = 45.46; CAER: 20.99; SpEmoC: 73.34
• MISA on MELD: Macro-F1 = 25.09; CAER: 32.82; SpEmoC: 50.78
• MulT on MELD: Macro-F1 = 25.09; CAER: 32.82; SpEmoC: 50.78
• Imbalance gap (Weighted F1 - Macro F1) on MELD/CAER usually 15-25 points, reduced to ≤0.65 on SpEmoC
• Cross-dataset transfer: Models trained on SpEmoC consistently outperform MELD/CAER training on minority classes Fear and Disgust with F1 >30 vs near zero under MELD/CAER transfer

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.18109.

Fig 1: Overview of SpEmoC construction and evaluation. Multimodal filtering and

Fig 3: Examples from SpEmoC showing variation in genre, lighting, color, and ex-

Fig 3 (page 5).

Fig 4 (page 5).

Fig 5 (page 5).

Fig 6 (page 5).

Fig 7 (page 5).

Fig 8 (page 5).

Limitations

• Demographic attributes such as age, gender, and ethnicity are inferred automatically from visuals and may contain errors due to makeup, lighting, or role-based disguises.
• The dataset focuses on English-language movies and TV series only, limiting cultural and language diversity.
• No explicit speaker diarization or active speaker detection; emotion labels correspond to dominant speaker segment which could miss multi-speaker emotions.
• Visual annotations are not directly used for labeling, relying on text and audio modalities primarily, possibly missing important visual cues.
• Human annotation, while substantial, still reports a moderate inter-annotator agreement (κ=0.62), indicating inherent label ambiguity.
• The dataset release is under a restricted EULA, potentially limiting broader public adoption or replication.

Open questions / follow-ons

• How well do models trained on SpEmoC perform on truly out-of-domain conversational data beyond movies or scripted TV shows?
• Can the multimodal labeling pipeline be adapted or extended to other languages, dialects, or culturally diverse emotional expressions?
• What is impact of incorporating speaker diarization or multi-speaker emotion annotations on model accuracy and fairness?
• How effective are different multimodal fusion architectures under low-resource, noisy, or real-time emotion recognition scenarios using SpEmoC?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focusing on multimodal emotion recognition, SpEmoC offers a significantly improved benchmark addressing key challenges such as class imbalance, multimodal alignment, and generalization stability. The strict movie-level data splits and balanced emotion classes reduce the risk of overfitting to dominant neutral or repeated content that can bias system behavior in real deployments. Deployment of emotion-aware bots or CAPTCHA systems can benefit from training on SpEmoC to improve recognition of minority emotional states (e.g., Fear, Disgust) often underrepresented but crucial for nuanced human interaction modeling. Moreover, the multimodal annotation pipeline combining text and audio modalities with human validation presents a scalable approach to create domain-specific emotion datasets with robust alignment, beneficial when building customized bot defenses that rely on affective cues from multimodal inputs. The insights about modality contribution and balanced data design can guide feature selection and evaluation protocols for affective CAPTCHA challenges that aim to be inclusive and resistant to manipulation by emotion-insensitive automated solvers.

Cite

@article{arxiv2607_18109,
  title={ SpEmoC: A Balanced Speaker-Segment Multimodal Emotion Benchmark },
  author={ Sania Bano and Shahzad Ahmad and Santosh Kumar Vipparthi and Sukalpa Chanda and Subrahmanyam Murala },
  journal={arXiv preprint arXiv:2607.18109},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.18109}
}

Read the full paper

• arXiv:2607.18109 (abstract)
• arXiv:2607.18109 (PDF)