dask / dask/distributed

Deadlock when serialising reference to queue

Open
#3,752 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In the code below, when using `x = Queue()` and submitting `dask.delayed(g)` (without `()`), both the client and scheduler seem to deadlock. This also applies to non-local clusters.

```py
if __name__ == '__main__':
import dask.distributed

client = dask.distributed.Client()

# OK:
#x = dask.distributed.Variable()
#x = dask.distributed.Semaphore()
#x = dask.distributed.Lock()

# Deadlock:
x = dask.distributed.Queue()

def f():
return 42

def g():
x
return 42

print(client.gather(client.compute(dask.delayed(f)())))
# => 42
print(client.gather(client.compute(dask.delayed(f))))
# =>
print(client.gather(client.compute(dask.delayed(g)())))
# => 42
print(client.gather(client.compute(dask.delayed(g))))
# => infinite hang in gather
```

Contributor guide

Open the contributing guide

Research direction

Start by running the provided reproducer with dask.distributed.Client, Queue, and dask.delayed, comparing Queue with Variable, Semaphore, and Lock. Trace the path triggered when a delayed reference to g is submitted without calling it. Done means the final gather returns 42 rather than hanging, for both local and non-local clusters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.