Cluster should support many worker types
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
The various Cluster objects often allow the user to provide specifications of a worker (cores, memory, software environment, ...) and then provides mechanisms around increasing and decreasing the number of workers.
However sometimes a dask deployment has a few different kinds of workers, for example machines with GPUs or high memory, or machines from a queue that is more or less expensive or reliable in some way.
This suggests that maybe the Cluster object should accept a list of worker pools, and provide common functionality around them.
Things like the widget are easy to scale to multiple pools. Adaptivity is a bit weirder.
cc @lesteve @jhamman (dask-jobqueue) @jcrist (dask-yarn) @jacobtomlinson (dask-kubernetes)
Credit for this thought goes to @lesteve
Contributor guide
Assessment
This issue has not been assessed yet.