dask / dask/distributed

Race condition in distributed.utils.All

Open
#4,356 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Sometimes, `Client.run` runs into a problem with the following traceback:

```
[...]
File "/home/alex/source/distributed/distributed/client.py", line 2521, in run
return self.sync(self._run, function, *args, **kwargs)
File "/home/alex/source/distributed/distributed/client.py", line 833, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/home/alex/source/distributed/distributed/utils.py", line 340, in sync
raise exc.with_traceback(tb)
File "/home/alex/source/distributed/distributed/utils.py", line 324, in f
result[0] = yield future
File "/home/alex/source/tornado/tornado/gen.py", line 762, in run
value = future.result()
File "/home/alex/source/pyenv/versions/3.6.12/lib/python3.6/asyncio/tasks.py", line 180, in _step
result = coro.send(None)
File "/home/alex/source/distributed/distributed/client.py", line 2454, in _run
if resp["status"] == "OK":
TypeError: 'NoneType' object is not subscriptable
```

I've only seen it on Python 3.6 for now, although that can also be coincidence. This was using distributed version 2.30.1. I've tracked this down to https://github.com/tornadoweb/tornado/issues/2034 - because of this, [`All`](https://github.com/dask/distributed/blob/master/distributed/utils.py#L212) doesn't always return values for all indices, but instead sometimes keeps the `None` from the initialization for some indices. I've already submitted a fix upstream, but maybe you are also interested in fixing this for current tornado versions in distributed. Cheers!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.