EpistasisLab / EpistasisLab/tpot

Does TPOT support memory when running dask.distributed?

Open
#1,228 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I wanted to use TPOT with
1. dask.distributed running multiple processes on the local machine
2. memory enabled to cache common transformation across processes (it's supposed to be multiprocessing-safe)

But I did two things that make me thing this mode of operation is not supported:
1. Setting breakpoint inside's joblib.Memory.cache() function - it only get's called to check if produced individual is valid (check_pipeline/_pre_test function)
2. Looking at the code that actually performs evaluation of individuals. Everything seems to happen inside dask_ml.model_selection._search.build_graph(). But the way it handles pipelines (if my analysis is correct) is to recursively extract all leaf transformers and estimators, turn them into Dask graph nodes and then, at the end, rebuild pipelines. No sklearn.Pipeline code appears to be executed (and that's where caching is implemented)

My questions are as follow:
1. Is my analysis correct and that mode is indeed unsupported?
2. What would be the easiest way to add this caching functionality?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.