Support port forwarding in SSHCluster
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Some users may have SSH access to a machine and want to use `SSHCluster` with it, but are not able to access ports other than 22 due to firewall restrictions.
It would be nice to be able to forward the scheduler ports locally through the SSH tunnel.
This would require:
- Add kwargs to `SSHCluster` to specify local port forwards
- Set up port forward in [scheduler object](https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L170) after connection is made ([docs](https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.SSHClientConnection.forward_remote_port))
- Set the scheduler address to the locally forwarded port [here](https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L206)
**Example**
```python
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
... ["host1", "host2", "host3", "host4"],
... forward_scheduler_port=8786,
... forward_dashboard_port=8787
... )
>>> client = Client(cluster)
```
There is an assumption here that the restrictive firewall rules would not limit worker-scheduler and worker-worker communication.
Contributor guide
Assessment
This issue has not been assessed yet.