WithinSessionEvaluation should behave differently for ERP-based paradigms
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 264
- Avg merge
- 1d 13m
- Merged PRs (30d)
- 23
Description
In many ERP-based datasets, the epochs aligned to a stimulus will overlap to a large extent, e.g. when considering a 0.5 second window after the stimulus, but the time between the onset of stimuli (SOA) was only 100ms. Consider the following example with 5 epochs, where T denotes train and V denotes validation.
onset T1 T2 V3 T4 T5
time 0 100 200 300 400
V3 is picked for validation purposes, however in 500ms epochs, the last 300ms of T1 contain the first 300ms of V3 and the first 400ms of T4 contain the last 400ms of V3, etc.
Note that this is usually not an issue in Motor Imagery / SSVEP as these epochs seldomly overlap.
In my fork of moabb I perform train / validation splits only on borders between runs (usually different EEG files and concluded short sections of recordings). However, some datasets do not have run-information, here we could either detect the borders based on event times, or if events only have sequence but no time information, just pick continuous sections of recording and omit X epochs at the border (according to SOA + epoch size).
The TimeSeriesSplit of sklearn could be an alternative but here each training-fold has different sizes. In my opinion for ERP/P300 paradigms it is not useful to train a classifier in WithinSessionEvaluation on e.g. 80% of the data, as these results are mostly useless for practical BCI usage.
I dont know whether to create a new WithinSessionEvaluationChronological or change the default behaviour of WithinSessionEvaluation (In my own fork of moabb I do the latter).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with WithinSessionEvaluation and compare the proposed TimeSeriesSplit approach with splits at run or recording boundaries. Resolve whether ERP paradigms should change the default or use a separate evaluation, including cases without run or event-time information; done means an agreed chronological split policy that prevents overlapping train and validation epochs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100