Race condition in distributed.utils.All
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Sometimes, `Client.run` runs into a problem with the following traceback:
```
[...]
File "/home/alex/source/distributed/distributed/client.py", line 2521, in run
return self.sync(self._run, function, *args, **kwargs)
File "/home/alex/source/distributed/distributed/client.py", line 833, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/home/alex/source/distributed/distributed/utils.py", line 340, in sync
raise exc.with_traceback(tb)
File "/home/alex/source/distributed/distributed/utils.py", line 324, in f
result[0] = yield future
File "/home/alex/source/tornado/tornado/gen.py", line 762, in run
value = future.result()
File "/home/alex/source/pyenv/versions/3.6.12/lib/python3.6/asyncio/tasks.py", line 180, in _step
result = coro.send(None)
File "/home/alex/source/distributed/distributed/client.py", line 2454, in _run
if resp["status"] == "OK":
TypeError: 'NoneType' object is not subscriptable
```
I've only seen it on Python 3.6 for now, although that can also be coincidence. This was using distributed version 2.30.1. I've tracked this down to https://github.com/tornadoweb/tornado/issues/2034 - because of this, [`All`](https://github.com/dask/distributed/blob/master/distributed/utils.py#L212) doesn't always return values for all indices, but instead sometimes keeps the `None` from the initialization for some indices. I've already submitted a fix upstream, but maybe you are also interested in fixing this for current tornado versions in distributed. Cheers!
Contributor guide
Assessment
This issue has not been assessed yet.