Flaky `distributed/tests/test_client.py::test_threadsafe_compute`
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
```
2022-11-18 05:04:47,523 - distributed.scheduler - ERROR - Error transitioning "('arange-sum-1960f5e08fa9c9689f5e9ef3be470377', 8)" from 'processing' to 'memory'
Traceback (most recent call last):
File "D:\a\distributed\distributed\distributed\scheduler.py", line 1842, in _transition
recommendations, client_msgs, worker_msgs = func(
File "D:\a\distributed\distributed\distributed\scheduler.py", line 2[359](https://github.com/dask/distributed/actions/runs/3494272750/jobs/5849912273#step:18:360), in transition_processing_memory
self._exit_processing_common(ts, recommendations)
File "D:\a\distributed\distributed\distributed\scheduler.py", line 3201, in _exit_processing_common
for qts in self._next_queued_tasks_for_worker(ws):
File "D:\a\distributed\distributed\distributed\scheduler.py", line 3218, in _next_queued_tasks_for_worker
assert qts.state == "queued", qts.state
AssertionError: forgotten
2022-11-18 05:04:47,523 - distributed.scheduler - INFO - Remove worker
2022-11-18 05:04:47,523 - distributed.core - INFO - Connection to tcp://127.0.0.1:52859 has been closed.
2022-11-18 05:04:47,523 - distributed.core - INFO - Removing comms to tcp://127.0.0.1:52864
2022-11-18 05:04:47,523 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:52864. Reason: worker-handle-scheduler-connection-broken
2022-11-18 05:04:47,820 - distributed.core - ERROR - forgotten
```
https://github.com/dask/distributed/actions/runs/3494272750/jobs/5849912273#step:18:352
This is weird. Maybe indicates an actual bug? I'm not sure how a task in the queue could be in state `forgotten`. `queued->released` removes the task from `scheduler.queue`. So any direct `queued->forgotten` transition would go through there first.
But given that the whole process dies, perhaps this is a red herring and something much deeper is broken, and this transition happens to be one manifestation of this thing breaking.
Contributor guide
Assessment
This issue has not been assessed yet.