dask / dask/distributed

SSHCluster multiple worker options

Open
#4,714 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently when using `SSHCluster` worker options are the same on all nodes. To a certain degree this limits the usage on clusters with heterogeneous nodes (e.g. some nodes have 4 cpu's while other might have more). So it would be nice if similar to the connect options different worker options could be specified for each node.

Additionally it would be nice to optionally be able to specify custom names for each node instead of the automatic generated number names.

The proposed changes should not change the current behaviour so should be backwards compatible and are mainly related to the lines:

https://github.com/dask/distributed/blob/74a9aff7e7429c4b2f12204cfa97aba1fca5aa29/distributed/deploy/ssh.py#L357-L366

Proposed change:
````Python
"options": {
"address": host,
"connect_options": connect_options
if isinstance(connect_options, dict)
else connect_options[i + 1],
"kwargs": worker_options
if isinstance(worker_options, dict)
else worker_options[i],
"worker_module": worker_module,
"remote_python": remote_python[i + 1]
if isinstance(remote_python, list)
else remote_python,
"name": names[i]
if isinstance(names, list)
else None,
},
````

`names` would have to be added as a new argument.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.