dask / dask/distributed

Scheduler deadlocked after stealing failed in `move_task_confirm`

Open
#8,787 0 comments 0 reactions 0 assignees View on GitHub
bug deadlock
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I've investigated a cluster that deadlocked after work-stealing failed in `move_task_confirm` with the following traceback:

```python
distributed.stealing - ERROR -
Traceback (most recent call last):
File "/opt/coiled/env/lib/python3.10/site-packages/distributed/stealing.py", line 380, in move_task_confirm
victim.remove_from_processing(ts)
File "/opt/coiled/env/lib/python3.10/site-packages/distributed/scheduler.py", line 771, in remove_from_processing
self.processing.remove(ts)
KeyError:
```

From what I understand, stealing has come pretty far in confirmation, i.e., it checked that the request is up-to-date, that the worker has indeed _confirmed_ the request (by checking the worker status), and checked whether the task is currently stealable.

After looking into this for a while, I have not been able to understand the root-cause of this, so I'm leaving this here in case this ever comes up again.

**Environment**:

- Dask version: `2024.7.1`
- Python version: `3.10.12`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.