dask / dask/distributed

KilledWorker deadlock

Open
#3,196 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

distributed 2.6.0
CPython 3.7.4
anaconda
Linux x64

Workers dying with SIGKILL randomly get the scheduler/client stuck.
This program gets consistently stuck for me in less than 2 minutes:

```python
import datetime
import os
import distributed
from distributed.scheduler import KilledWorker

def seppuku():
os.kill(os.getpid(), 9)

def main():
with distributed.LocalCluster(
n_workers=2, threads_per_worker=1, host="localhost:0", dashboard_address=None
) as cluster:
with distributed.Client(cluster, set_as_default=False) as client:
i = 0
while True:
fut = client.submit(seppuku, pure=False)
assert isinstance(fut.exception(), KilledWorker)
i+= 1
ts = datetime.datetime.now().strftime("%H:%M:%S")
print(f"{ts} Run {i} OK")

if __name__== "__main__":
main()

```
Output:
```
08:41:53 Run 425 OK
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
08:41:54 Run 426 OK <------ STUCK - hitting Ctrl+C after a couple of minutes
distributed.nanny - WARNING - Restarting worker
Traceback (most recent call last):
File "scratches/killedworker.py", line 27, in
main()
File "scratches/killedworker.py", line 20, in main
assert isinstance(fut.exception(), KilledWorker)
File "python3.7/site-packages/distributed/client.py", line 264, in exception
return self.client.sync(self._exception, callback_timeout=timeout, **kwargs)
File "python3.7/site-packages/distributed/client.py", line 762, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "python3.7/site-packages/distributed/utils.py", line 330, in sync
e.wait(10)
File "python3.7/threading.py", line 552, in wait
signaled = self._cond.wait(timeout)
File "python3.7/threading.py", line 300, in wait
gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

Process finished with exit code 1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.