dask / dask/distributed

Support `executor="processes"` and the like to Worker

Open
#5,319 3 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Being able to specify the worker's executor(s) is a neat feature that's probably under-used in part because it's currently a somewhat cumbersome API: you need to create and pass in the executor instance(s) you want to use.

On cloud deployments or using a spec cluster, this is awkward. You currently have write a worker plugin to do this (see https://youtu.be/vF2VItVU5zg?t=515). Instead, it would be nice if you could just do:
* `Worker(..., executor="processes")`
* `Worker(..., executor="threads")`
* `Worker(..., executor="concurrent.futures.ThreadPoolExecutor")`
* `Worker(..., executor="loky.ProcessPoolExecutor")`
* `Worker(..., executor={"default": "threads", "process": "processes"})`

If this is done, some other cleanup should follow:
- [ ] `LocalCluster(processes=True)` should create workers with `executor="processes"` instead of multiple single-threaded workers
- [ ] `dask-worker --nprocs` should do the same as above (xref https://github.com/dask/distributed/issues/2471)
- [ ] Perhaps `nthreads` should generally be renamed to `concurrency` or something? So `concurrency=` controls the pool size, `executor=` controls the pool type.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.