EpistasisLab / EpistasisLab/tpot
How to optimize Dask config for TPOT
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
I have a question regarding using Dask-jobqueue on HPC to parallelize TPOT. I have 10 nodes, with 47 cores each. I can either do:
```
cluster = LSFCluster(queue='corradin_long',
cores= 47,
walltime='100000:00',
memory='256GB',
death_timeout=600
cluster.adapt(minimum= 5, maximum=10)
```
or
```
cluster = LSFCluster(queue='corradin_long',
cores= 2,
walltime='100000:00',
memory='10GB',
death_timeout=600
cluster.adapt(minimum= 200, maximum=235)
```
According to Dask resources: [here](https://stackoverflow.com/questions/49406987/how-do-we-choose-nthreads-and-nprocs-per-worker-in-dask-distributed) and [here](http://jobqueue.dask.org/en/latest/configuration-setup.html):
> Using few processes and many threads per process is good if you are doing mostly numeric workloads, such as are common in Numpy, Pandas, and Scikit-Learn code, which is not affected by Python's Global Interpreter Lock (GIL).
> However, if you are spending most of your compute time manipulating Pure Python objects like strings or dictionaries then you may want to avoid GIL issues by having more processes with fewer threads each
So which configuration is best for TPOT ? In the [example video of using TPOT with Dask from Dask](https://www.youtube.com/watch?v=uyx9nBuOYQQ&t=265s), they used 40 workers and 80 cores (2 cores per worker) and it worked fine, but when I used 2 cores per worker with 200 workers it was very slow and results in a weird task graph. Although I couldn't see their cluster initialization so I might be doing it wrong (`processes` and `ncpus` keywords for the `LSFCluster` object initialization did not work for me) Sorry I'm new to both libraries and an inexperienced coder, so more layman terms explanation is greatly appreciated!
Also, I look at the conversation between dask and TPOT (#304) and while it's interesting, are there documentation on things to note/ limitations/ tips and tricks when using Dask with TPOT to achieve maximum efficiency? The [dask example binder](https://mybinder.org/v2/gh/dask/dask-examples/master?filepath=machine-learning%2Ftpot.ipynb) is a great starting point but it offers a very minimum example. I'd be willing to write up something to help people who uses the two libraries together.
Thanks so much for your work on this incredible library!!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.