Corrupted state machine: forgotten task in Scheduler.queued
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Just got a random failure in `test_client.py::test_threadsafe_compute `. The test may or may not be significant in reproducing.
No cluster dump as the test suite was killed by pytest-timeout.
https://github.com/dask/distributed/actions/runs/6420000156/job/17431105389?pr=8233
```
2023-10-05 14:10:53,031 - distributed.core - INFO - Connection to tcp://127.0.0.1:56258 has been closed.
2023-10-05 14:10:53,031 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:56269. Reason: worker-handle-scheduler-connection-broken
2023-10-05 14:10:53,031 - distributed.scheduler - INFO - Remove worker (stimulus_id='handle-worker-cleanup-1696515053.0367997')
2023-10-05 14:10:53,328 - distributed.core - ERROR - forgotten
Traceback (most recent call last):
File "D:\a\distributed\distributed\distributed\utils.py", line 801, in wrapper
return await func(*args, **kwargs)
File "D:\a\distributed\distributed\distributed\scheduler.py", line 4373, in add_worker
await self.handle_worker(comm, address)
File "D:\a\distributed\distributed\distributed\scheduler.py", line 5749, in handle_worker
await self.handle_stream(comm=comm, extra={"worker": worker})
File "D:\a\distributed\distributed\distributed\core.py", line 1002, in handle_stream
handler(**merge(extra, msg))
File "D:\a\distributed\distributed\distributed\scheduler.py", line 5615, in handle_task_finished
self.stimulus_queue_slots_maybe_opened(stimulus_id=stimulus_id)
File "D:\a\distributed\distributed\distributed\scheduler.py", line 4869, in stimulus_queue_slots_maybe_opened
assert qts.state == "queued", qts.state
AssertionError: forgotten
```
CC @fjetter @hendrikmakait
Contributor guide
Assessment
This issue has not been assessed yet.