ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions

1. Introduction: The Social Weight of Robot Errors

Robot errors—defined as deviations from intended or expected behavior—are more than technical malfunctions; they are social events. As autonomous systems transition from laboratory settings into human-centric environments, failures in navigation, manipulation, or social etiquette can disrupt the flow of interaction and severely diminish user trust. While robots often struggle to detect their own internal state transitions toward failure, humans naturally respond to these anomalies with observable social cues, such as surprise, confusion, or concern.

These “implicit human feedback” signals, expressed through facial expressions and body language, provide a valuable “out-of-band” signal for error detection. Operating independently of the robot’s internal diagnostic systems, these signals offer a redundant monitoring layer. This report summarizes the findings and methodologies of the ERR@HRI 3.0 Challenge, as reported by Parreira et al., which explores the use of multimodal perception to detect and anticipate robot failures through human affective responses.

2. The ERR@HRI 3.0 Challenge: A New Frontier in HRI Benchmarking

The ERR@HRI initiative serves as a platform for benchmarking multimodal machine learning models designed to detect robot errors via human behavioral signals. The 3.0 edition of the challenge introduced two major shifts in the research landscape:

• Raw Video Data: Moving beyond the pre-extracted features of previous years, ERR@HRI 3.0 released raw, non-anonymized webcam video. This enables end-to-end learning directly on visual data, allowing models to utilize the full depth of pixel-level information.
• Anticipatory Prediction: The challenge expanded beyond reactive detection (recognizing a failure that has already occurred) to include anticipatory prediction (identifying a likely failure before the outcome is revealed).

The authors chose to utilize “naturalistic” crowdsourced data rather than data from controlled laboratory settings. This approach exposes models to real-world variability, such as inconsistent lighting, diverse camera angles, varying participant positioning, and background noise. While these factors increase technical difficulty, the authors argue they are essential for developing robust systems capable of reliable real-world deployment.

3. Two Perspectives on Failure: The BAD and Bad Idea Datasets

The challenge provided participants with two distinct datasets to address both reactive and proactive error management.

Dataset Name	Participants	Temporal Focus	Label Type
Bystander Affect Detection (BAD)	45	During failure	Observed failure vs. Control
Bad Idea	29	Before failure	Participant-predicted outcome

The Bystander Affect Detection (BAD) Dataset This dataset contains webcam recordings of participants’ spontaneous reactions while watching videos of robot and human failures. It focuses on the reactive side of error detection. A significant characteristic of this dataset is its inherent class imbalance: 86.9% of the scenarios are failures, while only 13.1% are control (non-failure) scenarios. This reflects a design aimed at capturing a wide array of failure-induced affective responses, though it presents a significant hurdle for model training.

The Bad Idea Dataset In contrast, the Bad Idea dataset focuses on the moments before a failure occurs. Participants watched action scenarios that were cut off before the outcome was revealed and predicted if the scenario would end “well” or “poorly.” Crucially, the labels are based on the participants’ subjective predictions rather than the objective outcome of the scenario. In this track, models are tasked with predicting the human’s subjective sense of impending trouble, capturing the subtler cognitive signals associated with anticipation.

4. Tracking Success: Detection vs. Anticipation

The authors structured the challenge into three tracks to evaluate different aspects of error perception:

• Track 1: Bystander Reaction Detection: Models classified whether a participant was observing a failure or a control scenario based on facial reactions. To ensure models were capturing reactive signals rather than long-term context, the authors enforced a 2-second window constraint for video analysis.
• Track 2: Anticipatory Response Prediction: Models aimed to predict the participant’s subjective outcome prediction (well vs. poorly) from behavior recorded before the event concluded. The authors employed a max aggregation logic for this track, based on the principle that a brief, decisive anticipatory signal—such as a sharp intake of breath or a quick wince—is more informative for safety-critical intervention than a sustained, averaged reaction.
• Track 3: Cross-Dataset Generalization: An optional track exploring transfer learning between reactive and anticipatory contexts.

For Track 1, the primary metric was Macro F1 to account for the heavy class imbalance. Track 2 utilized AUC-ROC to reward models for confidently identifying brief but decisive signals. The authors also tracked “Earliest Detection Time” and “False Negative Rate” (FNR) to measure prediction consistency across video windows.

5. Benchmarking Performance: Baselines and Participant Results

The authors established baseline models for each track: BadNet (a convolutional architecture) for Track 1 and a fine-tuned ResNet-34 for Track 2.

The baseline results revealed the significant difficulty of the tasks:

• Track 1 (BAD): Achieved a Macro F1 of 0.502. The model struggled significantly with class imbalance, showing a high F1 score for failure detection (0.892) but a very low score for control scenarios (0.113). The baseline demonstrated an Earliest Detection Time of 8.8%.
• Track 2 (Bad Idea): Achieved an AUC-ROC of 0.564. The model showed more balanced per-class performance but struggled to extract a reliable signal, resulting in a much higher Earliest Detection Time of 35.6% and a higher FNR of 0.384.

The authors noted that while three participant teams surpassed these baselines, achieving high performance remains difficult due to the naturalistic variability of crowdsourced data. The disparity between the F1 scores in Track 1 highlights the risk of models defaulting to majority-class predictions in high-stakes failure scenarios.

6. The Generalization Gap: Why Real-World Detection is Hard

The paper identifies significant difficulties in cross-dataset generalization. A primary factor is the “subject-independent” testing protocol, where models are evaluated on participants they did not see during training. This necessitates that models learn generalizable features of human affect rather than over-fitting to specific individuals.

Furthermore, models struggle when moving from observed failures (BAD) to anticipated outcomes (Bad Idea). The facial signals associated with seeing a mistake occur in real-time appear to differ significantly from the subtler signals associated with predicting a future failure. This “generalization gap” suggests that a universal error-aware system must account for the distinct temporal phases of human reaction.

7. Key Takeaways for AI Safety and HRI

The ERR@HRI 3.0 Challenge provides three critical insights for the development of safety-critical embodied systems:

1. Transition to Raw Perceptual Data: The move toward end-to-end learning on raw visual data allows models to capture nuanced representations of human affect that pre-extracted features might miss. However, this requires architectures capable of filtering environmental noise.
2. Anticipatory Signals as Safety Redundancy: The ability to detect “bad ideas” before they manifest as failures provides a pathway for proactive intervention. These anticipatory signals serve as a redundant safety layer, offering an early-warning system that can trigger a “safe state” before an accident occurs.
3. Adversarial Robustness through “In-the-Wild” Testing: Benchmarking on “naturalistic” data—characterized by varying lighting, angles, and backgrounds—is essential for ensuring adversarial robustness. Systems that perform well in controlled labs often fail when exposed to the stochastic variability of real-world human environments.

8. Conclusion: The Path to Context-Aware Systems

The authors envision future iterations of the ERR@HRI initiative expanding to cover a wider range of interaction contexts and diverse robot embodiments. As noted in the paper, the ultimate goal is to move beyond reactive error correction toward context-aware systems. By effectively interpreting “out-of-band” implicit human feedback, autonomous robots can utilize human affect for error mitigation and preemptive safety intervention, leading to more adaptive and trustworthy partners in human-centric environments.

Read the full paper on arXiv · PDF