dask / dask/distributed

Handshake is incorrect for Client.gather(direct=False)

Open
#7,774 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

The design idea behind the connection handshake is to find a minimum common denominator on installed compression libraries and pickle protocol between the two peers on the connection.

The handshake about the pickle protocol is silly. If your minor python version is mismatched, you will experience a gazillion of unpickle failures anyway.

Then compression. If one of the two peers has lz4/snappy installed but the other doesn't, the connection will be uncompressed.
This doesn't work if the client does not have lz4 installed while both scheduler and worker do, and the client tries to retrieve a future with direct=False (which is the default, and frequently the only option due to security restrictions). In this case, the worker will perform the handshake *with the scheduler*, compress the data, and send it. Then, the scheduler will relay it as-is to the client.

# Reproducer
Terminal 1
```bash
$ mamba create -n test-cluster python=3.10 dask distributed
$ conda activate test-cluster
$ dask scheduler
```
Terminal 2
```bash
$ conda activate test-cluster
$ dask worker localhost:8786
```
Terminal 3
```bash
$ mamba create -n test-client python=3.10
$ conda activate test-client
$ pip install dask distributed
$ python
```
```python
>>> import distributed
>>> c = distributed.Client("localhost:8786")
distributed/client.py:1381: VersionMismatchWarning: Mismatched versions found

+---------+--------+-----------+---------+
| Package | Client | Scheduler | Workers |
+---------+--------+-----------+---------+
| lz4 | None | 4.3.2 | 4.3.2 |
| numpy | None | 1.24.2 | 1.24.2 |
| pandas | None | 2.0.0 | 2.0.0 |
+---------+--------+-----------+---------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
>>> x = client.submit(lambda: "x" * 11_000)
>>> x.result()
File "distributed/protocol/compression.py", line 204, in
compressions[c]["decompress"](frame)
KeyError: 'lz4'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.