dask / dask/distributed

How should `distributed` deal with mismatches between Python or package versions?

Open
#7,017 1 comment 0 reactions 0 assignees View on GitHub
discussion
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

There are currently a few issues with the way we handle mismatches between Python or package versions:

* #5731 highlights that `Client.get_versions(check=True)` does not work as suggested in the docstring and _never_ raises.
* This behavior has changed multiple times in the past (#3936, #4019)
* #7016 shows that using different Python versions between client and cluster may result in a `SystemError` or `segfault`.
* `Client._ensure_connected` contains dormant functionality to raise an `ImportError` in that is currently not triggered due to the implementation of `versions.error_message` (see point 1): https://github.com/dask/distributed/blob/0a0cfb8bd1fd68d3861bb7f722f9df9c342058d6/distributed/client.py#L1306-L1307
* `versions.error_message` does not consider the Python version of the scheduler (fix in #7018)

This raises the question how `distributed` should deal with version mismatches in general and how tolerant it should be, if at all. My current thoughts are that we should be strict about Python, required and optional packages again due to #7016 and the fragility of using different versions of `dask`-related packages. I don't know if we should necessarily raise per default in `Client._ensure_connected`, but we should at least make sure that we can raise using a flag (similar to `Client.get_versions(check=True)`).

**Notes**
> Cloudpickle can only be used to send objects between the exact same version of Python.

https://github.com/cloudpipe/cloudpickle

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.