EpistasisLab / EpistasisLab/tpot

How to optimize Dask config for TPOT

Open
#847 3 comments 0 reactions 0 assignees View on GitHub
bug need contributor question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Hi there,

I have a question regarding using Dask-jobqueue on HPC to parallelize TPOT. I have 10 nodes, with 47 cores each. I can either do:
```
cluster = LSFCluster(queue='corradin_long',
cores= 47,
walltime='100000:00',
memory='256GB',
death_timeout=600

cluster.adapt(minimum= 5, maximum=10)
```

or

```
cluster = LSFCluster(queue='corradin_long',
cores= 2,
walltime='100000:00',
memory='10GB',
death_timeout=600

cluster.adapt(minimum= 200, maximum=235)
```

According to Dask resources: [here](https://stackoverflow.com/questions/49406987/how-do-we-choose-nthreads-and-nprocs-per-worker-in-dask-distributed) and [here](http://jobqueue.dask.org/en/latest/configuration-setup.html):

> Using few processes and many threads per process is good if you are doing mostly numeric workloads, such as are common in Numpy, Pandas, and Scikit-Learn code, which is not affected by Python's Global Interpreter Lock (GIL).

> However, if you are spending most of your compute time manipulating Pure Python objects like strings or dictionaries then you may want to avoid GIL issues by having more processes with fewer threads each

So which configuration is best for TPOT ? In the [example video of using TPOT with Dask from Dask](https://www.youtube.com/watch?v=uyx9nBuOYQQ&t=265s), they used 40 workers and 80 cores (2 cores per worker) and it worked fine, but when I used 2 cores per worker with 200 workers it was very slow and results in a weird task graph. Although I couldn't see their cluster initialization so I might be doing it wrong (`processes` and `ncpus` keywords for the `LSFCluster` object initialization did not work for me) Sorry I'm new to both libraries and an inexperienced coder, so more layman terms explanation is greatly appreciated!

Also, I look at the conversation between dask and TPOT (#304) and while it's interesting, are there documentation on things to note/ limitations/ tips and tricks when using Dask with TPOT to achieve maximum efficiency? The [dask example binder](https://mybinder.org/v2/gh/dask/dask-examples/master?filepath=machine-learning%2Ftpot.ipynb) is a great starting point but it offers a very minimum example. I'd be willing to write up something to help people who uses the two libraries together.

Thanks so much for your work on this incredible library!!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.