ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions

Source: arXiv:2607.11570 · Published 2026-07-13 · By Maria Teresa Parreira, Micol Spitale, Maia Stiber, Shiye Cao, Amama Mahmood, Chien-Ming Huang et al.

TL;DR

The ERR@HRI 3.0 Challenge addresses the critical problem of multimodal error detection and anticipation in human-robot interaction (HRI), aiming to improve robot error recognition in realistic, naturalistic settings. Prior work in robot error detection typically focused on reactive detection using pre-extracted features in controlled lab environments, limiting robustness and generalizability. ERR@HRI 3.0 advances this by releasing two complementary raw video datasets collected via crowdsourcing under diverse environmental conditions: the Bystander Affect Detection (BAD) dataset capturing spontaneous facial reactions during observed failures, and the Bad Idea dataset capturing anticipatory facial responses before errors manifest. This allows end-to-end learning from pixels with greater ecological validity.

The challenge featured three tracks: reactive bystander error detection using BAD, anticipatory outcome prediction using Bad Idea, and an optional cross-dataset generalization. Three teams submitted valid models, and all outperformed provided baseline convolutional neural network methods. The BAD baseline obtained a macro F1 of 0.502 on a heavily imbalanced dataset, while the Bad Idea baseline achieved an AUC-ROC of 0.564 reflecting modest anticipatory signal extraction. The challenge highlights multimodal error detection spanning detection and anticipation phases, emphasizing the importance of context-aware, generalizable models for real-world HRI applications.

Key findings

• Baseline for Track 1 (BAD dataset) achieved macro F1 = 0.502, balanced accuracy = 0.502 with highly imbalanced labels (Failure 86.9%).
• Baseline for Track 2 (Bad Idea dataset) achieved AUC-ROC = 0.564, balanced accuracy = 0.561 on near-balanced classes.
• Track 1 baseline’s error class F1 score was 0.892 versus control class F1 of 0.113, reflecting majority class bias.
• Earliest Detection Time metric for baseline was 8.8% clip elapsed for Track 1 and 35.6% for Track 2, indicating difficulty in anticipatory prediction.
• Three participating teams submitted valid models surpassing baselines on at least one track.
• Datasets include 45 participants and 1,645 clips (BAD), and 29 participants with 865 clips (Bad Idea), both raw webcam video with naturalistic variability.
• BAD dataset clips averaged 15.5 seconds; Bad Idea clips averaged 1.95 seconds, highlighting temporal differences in detection tasks.
• Window size constraints: Track 1 window size capped at 2 seconds; Track 2 exempt due to brief clip length.

Threat model

The adversary considered is an implicit one of real-world variability in human facial reaction signals during robot errors or their anticipation; no active adversarial attacker is modeled. The challenge assumes no malicious manipulation of input data. The model must robustly detect error-related social signals despite naturalistic noise in video capture, lighting, participant position, and environment.

Methodology — deep read

1. Threat Model & Assumptions: The challenge focuses on detecting robot errors through human bystanders’ facial expressions either during (reactive detection) or before (anticipatory prediction) failures. The adversary in this context is not malicious, but the challenges aim to robustly identify error signals in unconstrained, naturalistic human reactions. There is no explicit adversarial attacker; rather, models must handle variability in real-world settings.

2. Data: Two crowdsourced datasets from Prolific participants were used:

• BAD (Bystander Affect Detection): 45 participants viewed 46 stimulus videos (40 failure scenarios, 6 controls), producing 1,645 raw webcam video clips (avg ~15.5s), labeled with binary Failure (1) vs Control (0) from stimulus ground truth. The dataset is highly imbalanced (87% failure class).
• Bad Idea: 29 participants watched 30 action scenarios truncated before outcomes, predicting whether the outcome would be good or poor. 865 clips (~1.95s avg) recorded anticipatory facial responses labeled by participant prediction (near-balanced 52.6% good vs 47.4% poor).

3. Architecture / Algorithm:

• Track 1 baseline used the BadNet convolutional network architecture trained on 5fps frames, with sigmoid activation, kernel size 8, dropout 0.7, batch size 32, learning rate 0.0001, trained for 350 epochs.
• Track 2 baseline fine-tuned a ResNet-34 pretrained on ImageNet on 30fps frames with dropout 0.7, batch size 64, learning rate 0.001, 100 epochs. Baseline models used weighted losses to address class imbalance. Participants were encouraged to develop end-to-end models operating directly on raw video data.

4. Training Regime: Baselines trained with inter-participant cross-validation holding out participants to ensure subject-independence. Final models trained on full train/validation split with held-out test participants for evaluation.

5. Evaluation Protocol:

• Tasks were binary classification at video level, aggregating sliding window predictions.
• Track 1 aggregated by majority vote across windows (max window size capped at 2s).
• Track 2 aggregated by max predicted probability across windows.
• Metrics were macro F1 and balanced accuracy (Track 1), AUC-ROC (Track 2), with temporal metrics Earliest Detection Time and False Negative Rate computed on positive class clips.
• Three submissions per track per team limited to avoid test set overfitting.

6. Reproducibility: All raw datasets, code, and baseline models are publicly available via GitHub repository, with a Data Use Agreement for accessing sensitive raw video data. This supports reproducibility and further benchmarking.

Example flow end-to-end: For Track 1, a participant watching a failure video generates a raw webcam recording of their facial expressions (~15.5s). This video is sampled at 5fps, segmented into sliding windows (each 5 frames with 2 frames slide). The BadNet CNN processes each window producing binary predictions. These are aggregated via majority vote to produce a final video-level failure/control classification, evaluated against ground truth labels. Model training uses weighted loss to compensate for class imbalance. Similar procedure with ResNet-34 for Track 2, but with 30fps frames and window size 10 frames.

Technical innovations

• Release of two complementary raw, non-anonymized webcam video datasets (BAD and Bad Idea) capturing naturalistic multimodal human reactions for error detection and anticipation in HRI, enabling end-to-end learning.
• Introduction of anticipation of errors in HRI through the Bad Idea dataset, shifting beyond reactive detection to proactive error prediction.
• Challenge design enforcing participant-independent splits and evaluation with temporal windowed predictions aggregated into video-level metrics providing robust generalization testing.
• Concurrent benchmarking of three tracks including cross-dataset transfer learning, encouraging exploration of generalizable, context-aware models in multimodal HRI error detection.

Datasets

• BAD (Bystander Affect Detection) — 45 participants, 1,645 raw webcam video clips (~25,527 seconds total) — crowdsourced via Prolific
• Bad Idea — 29 participants, 865 raw webcam video clips (~1,851 seconds total) — crowdsourced via Prolific

Baselines vs proposed

• Track 1 (BAD), BadNet baseline: macro F1 = 0.502, balanced accuracy = 0.502 vs submitted models: all surpassed baseline (exact team values not detailed).
• Track 1 baseline F1 for error class = 0.892, control class = 0.113, highlighting skew.
• Track 2 (Bad Idea), ResNet-34 fine-tuned baseline: AUC-ROC = 0.564, balanced accuracy = 0.561 vs submitted models: all surpassed baseline (team results not detailed).
• Earliest detection time Track 1 baseline = 8.8% clip elapsed; Track 2 = 35.6%, indicating anticipatory task difficulty.

Limitations

• Datasets remain relatively small with 45 and 29 participants respectively, limiting statistical power and diversity.
• Strong class imbalance in BAD dataset challenges models to identify minority control condition robustly.
• Baseline models provide only modest predictive performance, indicating task difficulty and leaving room for improvement.
• No adversarial robustness evaluation or testing against intentional attacker manipulation of facial cues is presented.
• External generalization beyond crowdsourced webcam data to robotics platforms in typical HRI settings remains to be validated.
• Temporal metrics for anticipatory prediction in very short clips may reflect consistency rather than actual early detection ability.

Open questions / follow-ons

• How to improve anticipatory error prediction performance with richer multimodal inputs beyond facial video, e.g., audio, body pose, or physiological signals?
• Can models trained on crowdsourced webcam data generalize reliably to in-the-wild deployments involving in-situ robot sensory streams?
• What role can temporal modeling architectures (e.g., transformers, spatiotemporal CNNs) play in better capturing unfolding error cues?
• How to mitigate class imbalance and model biases in datasets to improve minority class recognition and fairness?

Why it matters for bot defense

This work provides valuable insights into the challenges and methods of multimodal error detection in naturalistic settings, highlighting trade-offs of working with raw video data in the wild. For bot-defense engineers, the challenge datasets and baseline models illustrate how subtle human behavioral cues such as facial expressions can be leveraged to detect anomalous or unexpected events—paralleling how CAPTCHAs rely on human-specific signals to distinguish bots. Anticipatory detection using early facial cues also suggests future avenues for proactivity in bot-detection systems that seek to flag suspicious activity before full compromise or failure. However, the relatively modest baseline performance and dataset limitations underscore the difficulty of developing robust, generalized classifiers that handle variability in unconstrained environments—important cautionary lessons for deploying adaptive bot-detection under realistic web or service conditions.

Cite

@article{arxiv2607_11570,
  title={ ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions },
  author={ Maria Teresa Parreira and Micol Spitale and Maia Stiber and Shiye Cao and Amama Mahmood and Chien-Ming Huang and Hatice Gunes and Wendy Ju },
  journal={arXiv preprint arXiv:2607.11570},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.11570}
}

Read the full paper

• arXiv:2607.11570 (abstract)
• arXiv:2607.11570 (PDF)