dask / dask/distributed

Allow worker's internal Executor to uses processes

Open
#504 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I was thinking about scalability, and the issues I have encountered with IPython parallel, following recent discussions with [swift-lang](http://swift-lang.org). With IPython parallel, scaling is largely limited by engine connections to the central scheduler, which is largely similar to dask (though IPython makes several more connections per engine), and engines are one per concurrent task, which is typically one per core unless you are doing multi-threaded tasks at the user-level (via OpenMP, blas, Python threads, etc.). This makes it difficult to have one engine per large node, which would let you take the scaling pretty far.

One advantage distributed Workers have over IPython engines is that they can themselves be parallel entities thanks to the internal ThreadPoolExecutor. Of course, as is the way with Python, there is a class of Python-bound tasks where threads don't help as much as one would like, especially if you have a lot of them. Would it make sense to allow workers to use a ProcessPoolExecutor for this class of work? Are there downsides to this beyond the obvious extra serialization hop?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.