dask / dask/distributed

Support port forwarding in SSHCluster

Open
#3,881 5 comments 2 reactions 0 assignees View on GitHub
enhancement good second issue
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Some users may have SSH access to a machine and want to use `SSHCluster` with it, but are not able to access ports other than 22 due to firewall restrictions.

It would be nice to be able to forward the scheduler ports locally through the SSH tunnel.

This would require:
- Add kwargs to `SSHCluster` to specify local port forwards
- Set up port forward in [scheduler object](https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L170) after connection is made ([docs](https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.SSHClientConnection.forward_remote_port))
- Set the scheduler address to the locally forwarded port [here](https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L206)

**Example**

```python
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
... ["host1", "host2", "host3", "host4"],
... forward_scheduler_port=8786,
... forward_dashboard_port=8787
... )
>>> client = Client(cluster)
```

There is an assumption here that the restrictive firewall rules would not limit worker-scheduler and worker-worker communication.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.