NeuroTechX / NeuroTechX/moabb

WithinSessionEvaluation should behave differently for ERP-based paradigms

Open
#280 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement hard help wanted moabb
Dominant language
Python
Stars
1.1k
Forks
264
Avg merge
1d 13m
Merged PRs (30d)
23

Description

In many ERP-based datasets, the epochs aligned to a stimulus will overlap to a large extent, e.g. when considering a 0.5 second window after the stimulus, but the time between the onset of stimuli (SOA) was only 100ms. Consider the following example with 5 epochs, where T denotes train and V denotes validation.

onset T1   T2   V3   T4   T5

time   0  100  200  300  400

V3 is picked for validation purposes, however in 500ms epochs, the last 300ms of T1 contain the first 300ms of V3 and the first 400ms of T4 contain the last 400ms of V3, etc.

Note that this is usually not an issue in Motor Imagery / SSVEP as these epochs seldomly overlap.

In my fork of moabb I perform train / validation splits only on borders between runs (usually different EEG files and concluded short sections of recordings). However, some datasets do not have run-information, here we could either detect the borders based on event times, or if events only have sequence but no time information, just pick continuous sections of recording and omit X epochs at the border (according to SOA + epoch size).
The TimeSeriesSplit of sklearn could be an alternative but here each training-fold has different sizes. In my opinion for ERP/P300 paradigms it is not useful to train a classifier in WithinSessionEvaluation on e.g. 80% of the data, as these results are mostly useless for practical BCI usage.

I dont know whether to create a new WithinSessionEvaluationChronological or change the default behaviour of WithinSessionEvaluation (In my own fork of moabb I do the latter).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with WithinSessionEvaluation and compare the proposed TimeSeriesSplit approach with splits at run or recording boundaries. Resolve whether ERP paradigms should change the default or use a separate evaluation, including cases without run or event-time information; done means an agreed chronological split policy that prevents overlapping train and validation epochs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.