Keras_DeepConvNet.yml and Keras_ShallowConvNet.yml pipelines skipping datasets silently

Hello,

I am a master's student at University of Washington, Bothell campus and I am doing my thesis on reproducibility in the field of EEG motor imagery classification via machine learning. I am attempting to reproduce your benchmark results and I am running into some problems that I was hoping you could help with.

Here is the environment I'm running in. I am using a container, so I can change environments if needed:
moabb[deeplearning] version: 1.2.0
python version: 3.10.20
OS: Debian 12 (Bookworm)

Note that I used an older version of MOABB because my desired pipeline was removed from newer versions.

I've pasted my python file and two log files. One shows the pipeline failing at Zhou2016 because it doesn't have enough events. The other shows the output when Zhou2016 is commented out to prevent the failure. AlexMI, BNCI2014_001 and PhysionetMI all fail silently, and only results for Schirrmeister2017 and Weibo2014 are output.

I tried deleting the results folder and rerunning from scratch to rule out cached results.

moabb_braindecode.py

import moabb
from moabb import benchmark
from moabb.pipelines.utils import parse_pipelines_from_directory
import mne
mne.set_config("MNE_DATA", "/gscratch/comdata/raw_data/mne_data")

results_multiclass = benchmark(
    pipelines="../pipelines", 
    paradigms=["MotorImagery"],
    evaluations=["WithinSession"],
    include_datasets=[
	"AlexMI",
	"BNCI2014_001",
	"PhysionetMI",
	"Schirrmeister2017",
	"Weibo2014",
#	"Zhou2016",
    ]
)

# Print accuracy per pipeline per dataset
print(results_multiclass.groupby(["dataset", "pipeline"])["score"].mean())

# Save to CSV for later use
results_multiclass.to_csv("results_motor_imagery.csv", index=False)

Relevant portion of moabb_braindecode_cpu_34934528.out

Traceback (most recent call last):
  File "/gscratch/comdata/users/akoser/moabb_braindecode/moabb_braindecode.py", line 27, in <module>
    results_multiclass = benchmark(
  File "/usr/local/lib/python3.10/site-packages/moabb/benchmark.py", line 190, in benchmark
    paradigm_results = context.process(
  File "/usr/local/lib/python3.10/site-packages/moabb/evaluations/base.py", line 222, in process
    process_pipeline = self.paradigm.make_process_pipelines(
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/base.py", line 140, in make_process_pipelines
    epochs_pipeline = self._get_epochs_pipeline(return_epochs, return_raws, dataset)
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/base.py", line 389, in _get_epochs_pipeline
    ("events", self._get_events_pipeline(dataset)),
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/base.py", line 538, in _get_events_pipeline
    event_id = self.used_events(dataset)
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/motor_imagery.py", line 378, in used_events
    raise (
ValueError: Dataset Zhou2016 did not have enough events in None to run analysis

moabb_braindecode_gpu_35023148.out

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1777931155.227444   41268 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
I0000 00:00:1777931164.144223   41268 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 43487 MB memory:  -> device: 0, name: NVIDIA L40S, pci bus id: 0000:84:00.0, compute capability: 8.9
Choosing from all possible events
/usr/local/lib/python3.10/site-packages/moabb/datasets/fake.py:93: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_FAKEDATASET-IMAGERY-10-2--60-60--120-120--FAKE1-FAKE2-FAKE3--C3-CZ-C4_PATH"
  set_config(key, temp_dir)
/usr/local/lib/python3.10/site-packages/moabb/datasets/fake.py:93: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_FAKEVIRTUALREALITYDATASET-P300-21-1--60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60--120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120--TARGET-NONTARGET--C3-CZ-C4_PATH"
  set_config(key, temp_dir)

Schirrmeister2017-WithinSession:   0%|          | 0/14 [00:00<?, ?it/s]
Schirrmeister2017-WithinSession:   0%|          | 0/14 [00:00<?, ?it/s]

Weibo2014-WithinSession:   0%|          | 0/10 [00:00<?, ?it/s]
Weibo2014-WithinSession:   0%|          | 0/10 [00:00<?, ?it/s]
             dataset     evaluation              pipeline  avg score
0  Schirrmeister2017  WithinSession     Keras_DeepConvNet   0.327083
1  Schirrmeister2017  WithinSession  Keras_ShallowConvNet   0.811967
2          Weibo2014  WithinSession     Keras_DeepConvNet   0.191071
3          Weibo2014  WithinSession  Keras_ShallowConvNet   0.457143
dataset            pipeline            
Schirrmeister2017  Keras_DeepConvNet       0.327083
                   Keras_ShallowConvNet    0.811967
Weibo2014          Keras_DeepConvNet       0.191071
                   Keras_ShallowConvNet    0.457143
Name: score, dtype: float32

Start with Keras_DeepConvNet.yml and Keras_ShallowConvNet.yml, then reproduce the benchmark call in moabb_braindecode.py using the supplied environment and logs. Trace benchmark processing from moabb/benchmark.py and compare the requested datasets with the output; done means datasets are not skipped silently and the resulting behavior is explicit.