dask / dask/distributed

Corrupted state machine: forgotten task in Scheduler.queued

Open
#8,236 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Just got a random failure in `test_client.py::test_threadsafe_compute `. The test may or may not be significant in reproducing.
No cluster dump as the test suite was killed by pytest-timeout.

https://github.com/dask/distributed/actions/runs/6420000156/job/17431105389?pr=8233

```
2023-10-05 14:10:53,031 - distributed.core - INFO - Connection to tcp://127.0.0.1:56258 has been closed.
2023-10-05 14:10:53,031 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:56269. Reason: worker-handle-scheduler-connection-broken
2023-10-05 14:10:53,031 - distributed.scheduler - INFO - Remove worker (stimulus_id='handle-worker-cleanup-1696515053.0367997')
2023-10-05 14:10:53,328 - distributed.core - ERROR - forgotten
Traceback (most recent call last):
File "D:\a\distributed\distributed\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "D:\a\distributed\distributed\distributed\scheduler.py", line 4373, in add_worker
await self.handle_worker(comm, address)
File "D:\a\distributed\distributed\distributed\scheduler.py", line 5749, in handle_worker
await self.handle_stream(comm=comm, extra={"worker": worker})
File "D:\a\distributed\distributed\distributed\core.py", line 1002, in handle_stream
handler(**merge(extra, msg))
File "D:\a\distributed\distributed\distributed\scheduler.py", line 5615, in handle_task_finished
self.stimulus_queue_slots_maybe_opened(stimulus_id=stimulus_id)
File "D:\a\distributed\distributed\distributed\scheduler.py", line 4869, in stimulus_queue_slots_maybe_opened
assert qts.state == "queued", qts.state
AssertionError: forgotten
```

CC @fjetter @hendrikmakait

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.