Improve parallelisation of evaluations
Open
Nobody has claimed this yet.
- #486 by @bruAristimunha — closed without merging
enhancement
hard
moabb
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 264
- Avg merge
- 1d 13m
- Merged PRs (30d)
- 23
Description
After discussions at the braindecode code sprint and following up on #460, I think we should break down the evaluations into something like that:
class BaseEvaluation:
def __init__(
self,
...
n_nodes=1, # number of data chunks to load in memory in parallel.
n_jobs=1, # number of jobs per data chunk. One job fits one pipeline on one fold.
):
self.n_nodes = n_nodes
self.n_jobs = n_jobs
@abc.abstractmethod
def get_splits(self) -> list[dict, list[dict, list[int], list[int]]]:
"""
Return a list of pairs with:
* a dict of arguments to pass to self.paradigm.get_data to load a minimal data chunk
* a list of splits for this data chunk, i.e. triplets with:
- dict describing the split,
- list of train indices,
- list of test indices.
"""
pass
def process(self, pipelines):
splits = self.get_splits()
splits_todo = []
for datachunk_args, chunk_splits in splits:
missing_results = self.results.not_yet_computed(datachunk_args, chunk_splits, pipelines)
if missing_results:
splits_todo.append((datachunk_args, chunk_splits, missing_results))
Parallel(n_jobs=self.n_nodes)(delayed(self.process_datachunk)(pipelines, *args) for args in splits_todo)
return self.results.to_dataframe(pipelines=pipelines, ...)
def process_datachunk(self, pipelines, datachunk_args, chunk_splits, missing_results):
X, y, metadata = self.paradigm.get_data(**datachunk_args)
Parallel(n_jobs=self.n_jobs)(delayed(self.process_split)(p, X, y, metadata, *split) for split in chunk_splits for p in pipelines)
def process_split(self, clf, X, y, metadata, split_args, train_idx, test_idx):
clf = deepcopy(clf)
clf.fit(X[train_idx], y[train_idx])
score = clf.score(X[test_idx], y[test_idx])
self.results.add(datachunk_args, split_args, clf, score)
This would remove all the for loops we have in the different evaluations and allow for larger parallelisation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing evaluation implementations and the follow-up in #460; this issue does not name specific files or tests. Compare their current loops with the proposed BaseEvaluation methods, then verify that chunk-level and split-level parallelism preserve result collection and the final dataframe.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100