Daily Paper

OOD Detection for EEG-based Machine Learning in High-Risk Environments

The authors benchmark out-of-distribution detection methods for EEG models across two clinical prediction tasks to evaluate their downstream efficacy alongside uncertainty estimation.

arXiv:2608.17620 Empirical Study

Philipp Bomatter, Henry Gouk

eeg-analysisout-of-distribution-detectionuncertainty-estimationclinical-machine-learningdistribution-shift

1. Introduction: The Hidden Risks of Silent AI Failures in EEG Analysis

Machine learning (ML) models trained on electroencephalography (EEG) data present significant promise for automating clinical diagnostics, such as identifying brain pathology or classifying cognitive impairment. However, deploying these models in high-risk healthcare environments exposes critical vulnerabilities. EEG signals are extraordinarily sensitive to real-world distribution shifts. Minor hardware variations across clinics, subtle mismatches in preprocessing pipelines (such as filtering cutoff frequencies or reference electrode montages), and demographic shifts in patient populations can significantly alter signal characteristics. When deep neural networks encounter such out-of-distribution (OOD) data, they frequently suffer from catastrophic prediction failures while paradoxically outputting high confidence scores. In a clinical setting, these silent failures risk delivering invalid medical assessments without warning practitioners.

To address these vulnerabilities, a benchmark study by Philipp Bomatter and Henry Gouk (2026) systematically evaluates out-of-distribution detection methods for EEG models and tests their practical efficacy on downstream clinical prediction tasks. Rather than evaluating OOD detection strictly in artificial isolation—such as measuring a model’s ability to distinguish entirely different datasets or disjoint semantic classes—the authors assess OOD detection directly through its operational impact on downstream clinical diagnostic accuracy.

The central thesis of the study demonstrates that standard discriminative OOD detectors fail to identify OOD EEG signals caused by realistic covariate shifts, operating near chance level (AUROC≈0.50\text{AUROC} \approx 0.50). Conversely, generative models—specifically methods evaluating latent space representations such as Signal in the Noise (SITN)—successfully detect severe distribution shifts. Furthermore, the paper demonstrates that generative OOD detection and discriminative model uncertainty estimation capture fundamentally distinct failure modes. By integrating generative OOD detection with discriminative uncertainty estimation into a joint decision pipeline, practitioners can establish a robust safety net against silent predictive failures in safety-critical EEG applications.


2. Deconstructing the Benchmark: Controlled EEG Perturbations & Architecture Setup

To evaluate EEG model robustness under realistic clinical conditions, Bomatter and Gouk established a controlled benchmarking framework using two major clinical EEG datasets:

  • TUAB (Temple University Hospital Abnormal EEG Corpus): A curated dataset of over 2,300 participants annotated by board-certified neurologists for EEG normality versus pathology, evaluated as a binary classification task.
  • CAUEEG (Chung-Ang University Hospital EEG): A dataset of 1,122 participants featuring diagnostic annotations for normal cognition, mild cognitive impairment (MCI), or dementia, evaluated as a three-way classification task.

The study implemented a dual-model architecture trained exclusively on the unperturbed in-distribution (ID) training split of each dataset:

  1. Generative Model: An unsupervised UNet architecture (following Ho et al. and Dhariwal & Nichol) trained via a continuous-time flow-matching objective. Continuous-time flow matching constructs a linear path Xt=(1−t)Z+tXX_t = (1 - t)Z + tX for t∈[0,1]t \in [0, 1] between Gaussian noise Z∼N(0,I)Z \sim \mathcal{N}(0, \mathbf{I}) and data X∼PdataX \sim P_{\text{data}}, optimizing a neural network to learn the vector field mapping noise to the underlying probability distribution of unlabelled EEG signals. During evaluation, backward integration along the probability flow Ordinary Differential Equation (ODE) is executed using an ODE solver (Euler’s method with 100 timesteps).
  2. Discriminative Model: A supervised Temporal Convolutional Network (TCN) featuring 4 residual blocks, 64 filters per layer, and kernel size of 5 with dilated convolutions (1,2,4,81, 2, 4, 8), trained via cross-entropy loss to execute the specific downstream clinical classification task.

To evaluate model behavior under precise distribution shifts, the authors introduced a suite of five parametrizable EEG perturbations. Applying these transformations directly to the unperturbed ID test set created perfectly paired OOD test samples, providing the precise methodological isolation required to measure downstream clinical accuracy drops directly attributable to covariate shift.

Perturbation TypeDescription & Parameter Variations
Sampling Rate ShiftSignals are resampled to higher rates (125 Hz125\text{ Hz}, 150 Hz150\text{ Hz}, 200 Hz200\text{ Hz}) via bandlimited sinc interpolation relative to the baseline training rate of 100 Hz100\text{ Hz}, causing signals to appear artificially smoothed to models expecting lower sampling frequencies.
Channel Order PermutationThe positional layout of a random subset of channels in the 19-channel 10–20 system is permuted (swapping 10%, 50%, or 100% of channels), simulating physical electrode plugging errors into recording amplifiers.
Reference Scheme ChangesAverage-referenced training data is re-referenced to alternative clinical montages: Cz monopolar reference, linked-temporal reference (average of T7 and T8), or a longitudinal double-banana bipolar montage.
High-pass FilteringAdditional zero-phase Butterworth high-pass filtering is applied with increasing cutoff frequencies (2 Hz2\text{ Hz}, 4 Hz4\text{ Hz}, 8 Hz8\text{ Hz}) relative to the 1 Hz1\text{ Hz} baseline, progressively stripping low-frequency neural activity.
Low-pass FilteringAdditional low-pass filtering is applied with decreasing cutoff frequencies (45 Hz45\text{ Hz}, 30 Hz30\text{ Hz}, 15 Hz15\text{ Hz}) relative to the 50 Hz50\text{ Hz} baseline, progressively eliminating high-frequency neural activity.

3. Empirical Benchmark Results: Generative vs. Discriminative Paradigms

The benchmark evaluated OOD detection performance using the Area Under the Receiver Operating Characteristic (AUROC\text{AUROC}) curve across all perturbations and severity levels. The empirical findings reveal a dramatic divergence between discriminative and generative paradigms across both datasets.

Discriminative Failure Modes

Methods operating directly on the output logits or Softmax distributions of the supervised TCN—including Maximum Softmax Probability (MSP), Energy score, Out-of-Distribution Detector for Neural Networks (ODIN), and Activation Shaping (ASH)—proved largely incapable of detecting covariate distribution shifts. Across the majority of perturbation conditions on TUAB, discriminative methods performed near chance level (AUROC≈0.50\text{AUROC} \approx 0.50). Under sampling rate shifts on TUAB, MSP achieved AUROC\text{AUROC} scores of 0.517 at 125 Hz125\text{ Hz}, 0.457 at 150 Hz150\text{ Hz}, and 0.383 at 200 Hz200\text{ Hz}. Under a 100% channel shuffle on TUAB, MSP scored 0.526, ODIN scored 0.529, and ASH scored 0.460.

Penultimate Embedding Densities: The Mahalanobis Distance Middle Ground

To probe whether discriminative models possess OOD awareness prior to the final classification layer, the benchmark evaluated the Mahalanobis distance metric (Appendix B). This approach extracts penultimate-layer feature activations immediately following the TCN’s global average pooling layer, modeling class-conditional Gaussian distributions with a shared precision matrix fitted on ID validation data.

Mahalanobis distance represents a structural middle ground, yielding improved OOD detection under physical and spatial shifts. On TUAB, it achieved AUROC\text{AUROC} scores of 0.654 under 200 Hz200\text{ Hz} resampling, 0.775 under 100% channel shuffle, and 0.851 under Cz re-referencing (reaching 0.925 on CAUEEG). However, Mahalanobis distance exhibited a catastrophic vulnerability under high-pass filtering: its performance collapsed to an AUROC\text{AUROC} of 0.285 at 8 Hz8\text{ Hz} on TUAB and 0.153 on CAUEEG. Under frequency-attenuating shifts, feature embeddings systematically drift toward class centroids, causing Mahalanobis distance to perform substantially worse than random guessing.

Generative Superiority & Latent Space Goodness-of-Fit

In contrast, generative-model-based detection approaches—specifically Typicality, Density of State Estimation (DoSE), and Signal in the Noise (SITN)—achieved vastly superior OOD detection performance. Crucially, their detection capability scaled monotonically with perturbation severity.

Generative methods leverage explicit density modeling or latent space geometry. SITN operates by integrating continuous-time flow-matching ODE trajectories backward from data space XX to latent space ZZ using Euler integration over 100 timesteps. Rather than relying on scalar likelihoods, SITN performs a factorized goodness-of-fit test assessing whether the backward-integrated latent vector ZZ conforms to the isotropic standard Gaussian prior N(0,I)\mathcal{N}(0, \mathbf{I}).

On the TUAB dataset, SITN consistently emerged as the top-performing detector:

  • Re-referencing (Cz Monopolar): SITN achieved an AUROC\text{AUROC} of 0.998 (vs. MSP’s 0.517).
  • Sampling Rate Shift (200 Hz200\text{ Hz}): SITN achieved an AUROC\text{AUROC} of 0.989 (vs. MSP’s 0.383).
  • Low-pass Filtering (15 Hz15\text{ Hz}): SITN achieved an AUROC\text{AUROC} of 0.990 (vs. MSP’s 0.511).
  • Channel Shuffle (100%): SITN achieved an AUROC\text{AUROC} of 0.942 (vs. MSP’s 0.526).

Comparative evaluations on the CAUEEG benchmark (Appendix A) corroborated these dynamics while introducing subtle paradigm shifts. On CAUEEG, DoSE (specifically DoSEKDE\text{DoSE}_{\text{KDE}}, which fits kernel density estimators over likelihood, latent prior probability, and Jacobian determinant statistics) narrowly surpassed SITN under specific montages:

  • Re-referencing (Cz, T7/T8, Bipolar): DoSE achieved a perfect AUROC\text{AUROC} of 1.000 across all three montages on CAUEEG, slightly outperforming SITN (0.997).
  • Low-pass Filtering (15 Hz15\text{ Hz}): DoSE achieved an AUROC\text{AUROC} of 0.998 on CAUEEG, compared to SITN’s 0.994.
  • Sampling Rate Shift (200 Hz200\text{ Hz}): SITN retained leadership on CAUEEG with an AUROC\text{AUROC} of 0.992 (vs. DoSE’s 0.965 and MSP’s 0.470).
  • High-pass Filtering (8 Hz8\text{ Hz}): SITN achieved an AUROC\text{AUROC} of 0.899 on CAUEEG (vs. DoSE’s 0.626 and MSP’s 0.518).

The Likelihood Paradox in High-Dimensional EEG Space

The benchmark highlights a critical failure mode regarding raw generative Log-likelihood. On the TUAB 200 Hz200\text{ Hz} resampling shift, raw log-likelihood produced an AUROC\text{AUROC} of 0.143 (and 0.056 on CAUEEG), meaning OOD samples were assigned systematically higher probability densities than valid in-distribution data.

This empirical result confirms that EEG analysis suffers from the high-dimensional likelihood paradox previously identified in computer vision (e.g., Serrà et al.). Raw log-likelihood measures signal complexity rather than domain validity. Artificially smoothed EEG signals (such as those resampled to 200 Hz200\text{ Hz} or flat channels) occupy high-density probability volume, yielding inflated log-likelihoods. Conversely, high-amplitude physiological or electrode artifacts yield extremely low log-likelihoods regardless of whether they originate from in-distribution recordings. Specialized metrics like SITN and DoSE bypass this paradox by evaluating factorized latent prior structure and probability mass distributions rather than unnormalized volume densities.


4. Conceptual Breakthrough: Disentangling True OOD Detection from Model Uncertainty

By evaluating downstream classification accuracy across percentile bins of OOD detector scores, Bomatter and Gouk resolved a persistent ambiguity in machine learning safety research: the conflation of true OOD detection with model uncertainty estimation.

                    ┌─────────────────────────────────────────┐
                    │            Input EEG Signal             │
                    └────────────────────┬────────────────────┘
                                         │
                                         ▼
                    ┌─────────────────────────────────────────┐
                    │     Stage 1: Generative OOD Filter      │
                    │                 (SITN)                  │
                    └─────────┬─────────────────────┬─────────┘
                              │                     │
                  Fails Check │                     │ Passes Check
             (Unfamiliar Data)│                     │ (Familiar Data)
                              ▼                     ▼
                   ┌────────────────────┐ ┌───────────────────┐
                   │ Abstain / Defer to │ │ Stage 2: ID Model │
                   │    Human Expert    │ │ Uncertainty (MSP) │
                   └────────────────────┘ └─┬───────────────┬─┘
                                            │               │
                             High Uncertainty│               │ Low Uncertainty
                             (Ambiguous ID) │               │ (Confident ID)
                                            ▼               ▼
                                 ┌──────────────────┐ ┌───────────┐
                                 │ Abstain / Defer  │ │ Predict   │
                                 │ to Human Expert  │ │ Output    │
                                 └──────────────────┘ └───────────┘

True OOD Detection (Generative Paradigm)

Generative methods like SITN evaluate data familiarity. They measure whether an input sample resides within the support of the training data distribution. When presented with severe covariate shifts (such as 200 Hz200\text{ Hz} resampling), SITN isolates OOD samples into extreme score percentiles. Downstream evaluations confirm that discriminative TCN accuracy on these flagged OOD samples drops to chance level (50.0%50.0\% on TUAB binary classification, 33.3%33.3\% on CAUEEG three-way classification). True OOD detection serves as an external safety filter, identifying inputs where the classifier lacks valid exposure.

Model Uncertainty Estimation & The Overconfidence Anomaly

Discriminative methods like MSP evaluate predictive ambiguity. Although MSP fails to separate covariate-shifted data from ID data, it exhibits strong predictive calibration across unperturbed in-distribution samples. Within familiar ID data, lower MSP scores correlate with reduced accuracy on difficult, boundary-adjacent patient cases.

However, the benchmark uncovered a severe diagnostic failure mode within discriminative metrics: the Overconfidence Anomaly. When evaluating MSP across score percentiles on combined ID and OOD data (TUAB 200 Hz200\text{ Hz} resampling), classification accuracy increases monotonically alongside confidence up to the 99.9th percentile (reaching 85%85\% accuracy). But at the extreme 99.9–100th percentile bin—where the model asserts maximum certainty—accuracy plummets catastrophically to 47.0%47.0\% (worse than chance level).

Even when restricted strictly to in-distribution ID data, MSP accuracy drops from 96.0%96.0\% at the 99.9th percentile down to 48.0%48.0\% in the top 0.1% confidence bin. This anomaly occurs because discriminative neural networks assign maximum Softmax confidence to specific corrupted, highly filtered, or artifact-laden signals that project far along extreme decision-boundary trajectories. Discriminative models fail most catastrophically precisely when asserting maximum confidence. In contrast, SITN exhibits monotonic reliability, maintaining uniform high accuracy across accepted score percentiles without overconfidence collapses.

These findings validate the theoretical framework of Li et al. (2025): standard supervised discriminative classifiers “answer the wrong question for OOD detection.” Repurposing classification confidence as an OOD metric is fundamentally flawed for detecting covariate distribution shifts, yet discriminative confidence remains valuable as an internal proxy for model uncertainty on familiar data.


5. The Dual Safety Net: Combining OOD Detection and Uncertainty Estimation

Because generative OOD detection and discriminative model uncertainty capture complementary failure modes, Bomatter and Gouk proposed a dual-stage rejection architecture for clinical deployment.

Under this framework, incoming EEG recordings pass through a two-stage sequential evaluation. First, a generative OOD detector (SITN, calibrated on ID validation data to enforce a fixed 1.0%1.0\% false positive rate) screens for data familiarity. Samples passing the OOD check proceed to a discriminative uncertainty estimator (MSP, calibrated via isotonic regression on ID validation data). If the calibrated probability of correct classification falls below a defined clinical threshold (e.g., 60.0%60.0\%), the system abstains. The clinical model outputs an automated prediction only if an input satisfies both safety checks; otherwise, it defers to a human neurologist.

The operational performance of this dual safety net is detailed in Table 2 for the TUAB dataset under a severe 200 Hz200\text{ Hz} resampling shift (evaluated on an experimental test pool comprising an equal 50/50 mix of unperturbed ID samples and perturbed OOD samples).

Rejection System StateID Only CoverageID Only AccuracyCombined (ID + OOD) CoverageCombined (ID + OOD) Accuracy
Baseline (No Rejection)100.0%80.5%100.0%67.5%
SITN Only (OOD Detection)98.5%80.8%49.5%80.7%
MSP Only (Model Uncertainty)88.1%83.8%89.2%69.3%
Combined (SITN + MSP)86.9%84.1%43.7%84.1%

Granular Coverage & Operational Breakdown

Understanding the coverage metrics requires tracing sample movement through the dual pipeline across the 50/50 ID+OOD test pool:

  • Unprotected Baseline: Without rejection, encountering the 200 Hz200\text{ Hz} resampling shift causes overall diagnostic accuracy to collapse from 80.5%80.5\% (ID only) down to 67.5%67.5\% across the combined pool.
  • MSP Single Safety Net: Applying MSP rejection alone filters out ambiguous samples, improving ID-only accuracy to 83.8%83.8\% (at 88.1%88.1\% ID coverage). However, because MSP cannot recognize covariate shifts, it fails to reject resampled OOD inputs. The accepted pool remains heavily contaminated with OOD samples, yielding a poor combined accuracy of 69.3%69.3\% despite rejecting over 10%10\% of total inputs.
  • SITN Single Safety Net: SITN correctly evaluates data familiarity. Across the 50/50 pool, SITN flags and rejects ≈50.5%\approx 50.5\% of all samples—successfully rejecting 99.0%99.0\% of OOD samples while maintaining a tight 1.5%1.5\% false positive rate on ID data (98.5%98.5\% ID coverage). This isolates an accepted pool of ≈49.5%\approx 49.5\% total coverage (consisting almost exclusively of valid ID signals), stabilizing combined accuracy at 80.7%80.7\%.
  • Combined Dual Safety Net: The hybrid pipeline achieves superior performance. SITN first eliminates the unfamiliar OOD sample population, reducing the pool to ≈49.5%\approx 49.5\% coverage. MSP then probes this remaining valid ID population, rejecting an additional ≈5.8%\approx 5.8\% of total samples that exhibit high internal decision ambiguity. This yields a final accepted coverage of 43.7%43.7\% with an overall clinical diagnostic accuracy of 84.1%84.1\%.

6. Conclusion & Key Takeaways for AI Safety Researchers

The benchmarking study by Bomatter and Gouk demonstrates that evaluating out-of-distribution detection methods in isolation—without measuring downstream task performance—obscures clinical utility. By measuring OOD detection directly alongside downstream diagnostic accuracy, the authors clarify how distribution shifts trigger silent model failures and demonstrate how layered architectures mitigate these risks.

Key takeaways for machine learning reliability and clinical AI safety include:

  • Generative Primacy for True OOD Detection: Generative models—specifically latent space goodness-of-fit methods like SITN and multi-statistic estimators like DoSE—represent the current state-of-the-art for detecting covariate distribution shifts in EEG signals, vastly outperforming discriminative methods.
  • Functional Separation of Safety Signals: AI safety architectures must explicitly decouple OOD detection (which evaluates data familiarity relative to the training manifold) from model uncertainty estimation (which quantifies decision ambiguity on familiar data).
  • Layered Defense Implementation: High-risk clinical deployments require hybrid rejection pipelines. Sequential pipelines combining generative OOD filters with calibrated discriminative uncertainty thresholds maximize classification accuracy and eliminate overconfident mispredictions.
  • Benchmark Reform in ML Safety: Benchmarking protocols must move beyond evaluating purely semantic class shifts in isolation. Future ML safety evaluations must prioritize realistic covariate distribution shifts evaluated directly against downstream task performance.

Read the full paper on arXiv · PDF