dask / dask/distributed

SSHCluster - localhost to localhost - The command line is too long

Open
#8,138 3 comments 3 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I am trying to run a multi-node, multi-host SSH cluster on Windows. I simplified it, for now, attempting to run both the scheduler and the workers on localhost. Based on the Dask documentation instructions, I setup public key SSH access, in this case, from localhost to localhost. Encountered this [issue](https://github.com/dask/distributed/issues/5411) and fixed it by the recommended fix in the same link. Then encountered the next issue, which has to do with trying to run a command which is over the character limit imposed by Windows.
```
set_env = "set DASK_INTERNAL_INHERIT_CONFIG={} &&".format(
dask.config.serialize(dask.config.global_config)
)
```
The above line from the "distributed\deploy\ssh.py", generates a string of 9000+ chars. Which seems to be a [problem](https://learn.microsoft.com/en-US/troubleshoot/windows-client/shell-experience/command-line-string-limitation).

The next line of code creates the command "cmd", and the following line starts the process:
`self.proc = await self.connection.create_process(cmd)`

and the below line extracts this error - 'The command line is too long.\r\n':
`line = await self.proc.stderr.readline()`

In an attempt to reduce the size of the serialized config, I have tried removing the Kubernetes key from the dask.config.global_config, and re-adding it with an empty dict as value, thinking I should not need Kubernetes, since I am using the SSHCluster and not KubeCluster. When serializing the config, the length is less than the limit, and sure enough, I seem to get past the 'The command line is too long' error but get stuck with the below error instead:

> 2023-08-28 21:10:06,883 - distributed.deploy.ssh - INFO - raise JSONDecodeError("Expecting value", s, err.value) from None
> 2023-08-28 21:10:06,883 - distributed.deploy.ssh - INFO - json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I am using Windows right now, and am considering installing a Linux VM to try this out. Was wondering if anyone has had this issue with Windows and what can be done to workaround it?

This is the code I am using in the main module:

```
import dask
from dask.distributed import Client, SSHCluster
cluster = SSHCluster(["localhost", "localhost"],
connect_options={"known_hosts": None},
worker_options={"n_workers": 10},
scheduler_options={"port": 0, "dashboard_address": ":8797"})
client = Client(cluster)
```

Environment:

- Dask version 2023.8.1
- Python version 3.11.2
- OS: Windows 10
- Installed via Pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.