Queue and Variable assume the scheduler address is identical across workers
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
When using a distributed cluster wherein different workers may have a different scheduler address (because of proxying / network configuration), one runs into errors when attempting to deserialize a distributed.Queue or Variable as these data structures include an attribute which indicates the scheduler address on the worker on which it was created.
See https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/queues.py#L281, https://github.com/dask/distributed/blob/1d7640b0172febf9ceef37c2c31241c66ac165eb/distributed/variable.py#L230.
As the scheduler address may be different on the worker on which it is de-serialized, the de-serialization would leave the object in an unusable state.
Traceback:
```
distributed.worker - WARNING - Could not deserialize task Traceback (most recent call last): File "/workspace/venv/lib/python3.6/site-packages/distributed/worker.py", line 2441, in _maybe_deserialize_task function, args, kwargs = _deserialize(*self.tasks[key]) File "/workspace/venv/lib/python3.6/site-packages/distributed/worker.py", line 3277, in _deserialize args = pickle.loads(args) File "/workspace/venv/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 66, in loads return pickle.loads(x) File "/workspace/venv/lib/python3.6/site-packages/distributed/variable.py", line 233, in __setstate__ client = get_client(address) File "/workspace/venv/lib/python3.6/site-packages/distributed/worker.py", line 3129, in get_client return Client(address, timeout=timeout) File "/workspace/venv/lib/python3.6/site-packages/distributed/client.py", line 674, in __init__ self.connection_args = self.security.get_connection_args("client") File "/workspace/venv/lib/python3.6/site-packages/distributed/security.py", line 222, in get_connection_args "ssl_context": self._get_tls_context(tls, ssl.Purpose.SERVER_AUTH), File "/workspace/venv/lib/python3.6/site-packages/distributed/security.py", line 201, in _get_tls_context ctx.load_cert_chain(cert_path, key_path) FileNotFoundError: [Errno 2] No such file or directory
```
**What you expected to happen**:
Contrast with https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/semaphore.py#L502 .
For the record, this came up in an internal discussion involving @jonashaag , @crepererum and @fjetter . Credit goes to them
Contributor guide
Assessment
This issue has not been assessed yet.