Support `executor="processes"` and the like to Worker
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Being able to specify the worker's executor(s) is a neat feature that's probably under-used in part because it's currently a somewhat cumbersome API: you need to create and pass in the executor instance(s) you want to use.
On cloud deployments or using a spec cluster, this is awkward. You currently have write a worker plugin to do this (see https://youtu.be/vF2VItVU5zg?t=515). Instead, it would be nice if you could just do:
* `Worker(..., executor="processes")`
* `Worker(..., executor="threads")`
* `Worker(..., executor="concurrent.futures.ThreadPoolExecutor")`
* `Worker(..., executor="loky.ProcessPoolExecutor")`
* `Worker(..., executor={"default": "threads", "process": "processes"})`
If this is done, some other cleanup should follow:
- [ ] `LocalCluster(processes=True)` should create workers with `executor="processes"` instead of multiple single-threaded workers
- [ ] `dask-worker --nprocs` should do the same as above (xref https://github.com/dask/distributed/issues/2471)
- [ ] Perhaps `nthreads` should generally be renamed to `concurrency` or something? So `concurrency=` controls the pool size, `executor=` controls the pool type.
Contributor guide
Assessment
This issue has not been assessed yet.