Add new worker specs in groups
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Currently a subclass of SpecCluster can generate new workers for the spec with `SpecCluster.new_worker_spec`, which returns a name and a worker configuration. This is typically called by scale.
```python
while len(cluster.worker_spec) < target_workers:
name, spec = cluster.new_worker_spec()
cluster.worker_spec[name] = spec
```
However, in both HPC and GPU cases we want/need to create workers in groups. For this reason it might be nice to change the API here to return a dictionary of new workers
```python
while len(cluster.worker_spec) < target_workers:
d = cluster.new_worker_spec()
cluster.worker_spec.update(d)
```
See https://github.com/dask/dask-jobqueue/pull/307#issuecomment-525415832 for motivation. This also comes up with GPU work, where we want to add a new node and have one worker per GPU on that node.
Contributor guide
Assessment
This issue has not been assessed yet.