Allow worker's internal Executor to uses processes
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I was thinking about scalability, and the issues I have encountered with IPython parallel, following recent discussions with [swift-lang](http://swift-lang.org). With IPython parallel, scaling is largely limited by engine connections to the central scheduler, which is largely similar to dask (though IPython makes several more connections per engine), and engines are one per concurrent task, which is typically one per core unless you are doing multi-threaded tasks at the user-level (via OpenMP, blas, Python threads, etc.). This makes it difficult to have one engine per large node, which would let you take the scaling pretty far.
One advantage distributed Workers have over IPython engines is that they can themselves be parallel entities thanks to the internal ThreadPoolExecutor. Of course, as is the way with Python, there is a class of Python-bound tasks where threads don't help as much as one would like, especially if you have a lot of them. Would it make sense to allow workers to use a ProcessPoolExecutor for this class of work? Are there downsides to this beyond the obvious extra serialization hop?
Contributor guide
Assessment
This issue has not been assessed yet.