SSHCluster multiple worker options
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Currently when using `SSHCluster` worker options are the same on all nodes. To a certain degree this limits the usage on clusters with heterogeneous nodes (e.g. some nodes have 4 cpu's while other might have more). So it would be nice if similar to the connect options different worker options could be specified for each node.
Additionally it would be nice to optionally be able to specify custom names for each node instead of the automatic generated number names.
The proposed changes should not change the current behaviour so should be backwards compatible and are mainly related to the lines:
https://github.com/dask/distributed/blob/74a9aff7e7429c4b2f12204cfa97aba1fca5aa29/distributed/deploy/ssh.py#L357-L366
Proposed change:
````Python
"options": {
"address": host,
"connect_options": connect_options
if isinstance(connect_options, dict)
else connect_options[i + 1],
"kwargs": worker_options
if isinstance(worker_options, dict)
else worker_options[i],
"worker_module": worker_module,
"remote_python": remote_python[i + 1]
if isinstance(remote_python, list)
else remote_python,
"name": names[i]
if isinstance(names, list)
else None,
},
````
`names` would have to be added as a new argument.
Contributor guide
Assessment
This issue has not been assessed yet.