dask / dask/distributed

Add new worker specs in groups

Open
#2,999 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently a subclass of SpecCluster can generate new workers for the spec with `SpecCluster.new_worker_spec`, which returns a name and a worker configuration. This is typically called by scale.

```python
while len(cluster.worker_spec) < target_workers:
name, spec = cluster.new_worker_spec()
cluster.worker_spec[name] = spec
```

However, in both HPC and GPU cases we want/need to create workers in groups. For this reason it might be nice to change the API here to return a dictionary of new workers

```python
while len(cluster.worker_spec) < target_workers:
d = cluster.new_worker_spec()
cluster.worker_spec.update(d)
```

See https://github.com/dask/dask-jobqueue/pull/307#issuecomment-525415832 for motivation. This also comes up with GPU work, where we want to add a new node and have one worker per GPU on that node.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.