New CrossSubjecEvaluation that supports transfer learning methods
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 264
- Avg merge
- 1d 13m
- Merged PRs (30d)
- 23
Description
I am proposing a new class called CrossSubjectTargetAwareEvaluation that extends cross-subject evaluation to pipelines that require subject metadata or optional unlabeled/labeled target-subject calibration data. The existing CrossSubjectEvaluation becomes one of 6 options that can be configured.
All this is motivated by the large number of papers on Transfer Learning. So we need:
- a way to provide not merged subjects, but all subjects separately, so that transfer learning methods can work. Let's call this "metadata-aware".
- allow a well defined controlled access to the held out subject (HOS), also called the test subject or heldout target
I myself am working on a transfer learning cross subject method and all this is motivated by real needs.
Currently in my code I have defined:
class CsMode(Enum):
HOS_SOURCE_ONLY_TRIALWISE = auto() # Source-only training; each held-out target trial is predicted independently.
HOS_SOURCE_ONLY_BLOCKWISE = auto() # Source-only training; held-out target trials are predicted as a block, matching standard MOABB behavior.
HOS_UNLABELED_20P = auto() # First 20% of held-out target trials are used unlabeled for adaptation; remaining 80% are evaluated.
HOS_UNLABELED_50P = auto() # First 50% of held-out target trials are used unlabeled for adaptation; remaining 50% are evaluated.
HOS_UNLABELED_100P = auto() # All held-out target trials are used unlabeled for transductive adaptation and also evaluated.
HOS_LABELED_20P = auto() # First 20% of held-out target trials are used with labels for supervised calibration; remaining 80% are evaluated.
This will allow people to start comparing their methods clearly. Some will focus on HOS_UNLABELED_20P, others on HOS_SOURCE_ONLY_TRIALWISE, but the results will become comparable. I work on HOS_SOURCE_ONLY_TRIALWISE.
So the objective is:
- preserve MOABB cache/result/save mechanics
- support old pipelines
- support final metadata-aware classifiers (heldout subject aware classifiers)
- support intermediate metadata-aware transformers (heldout subject aware transformers)
- transform X_target_unlabeled through earlier steps
Notes:
- HOS_SOURCE_ONLY_BLOCKWISE predicts the target block after source-only training (the heldout subject is not provided during training), while HOS_UNLABELED_100P explicitly uses the entire unlabeled target block during fit/adaptation before predicting it.
I will provide code in a PR later. As you can imagine, it is a big feature :)
I think the best strategy is to have CrossSubjectTargetAwareEvaluation along the CrossSubjectEvaluation. And allow people to test it for a while.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing CrossSubjectEvaluation and its cache, result, and save mechanics, then compare them with the proposed CrossSubjectTargetAwareEvaluation modes. Define how metadata, held-out target access, and unlabeled or labeled calibration data flow through pipelines while preserving old pipelines. Done means the new evaluation supports the stated modes and remains comparable with existing cross-subject results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100