dask / dask/distributed

Flaky `distributed/tests/test_client.py::test_threadsafe_compute`

Open
#7,336 0 comments 0 reactions 0 assignees View on GitHub
flaky test
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

```
2022-11-18 05:04:47,523 - distributed.scheduler - ERROR - Error transitioning "('arange-sum-1960f5e08fa9c9689f5e9ef3be470377', 8)" from 'processing' to 'memory'

Traceback (most recent call last):

File "D:\a\distributed\distributed\distributed\scheduler.py", line 1842, in _transition

recommendations, client_msgs, worker_msgs = func(

File "D:\a\distributed\distributed\distributed\scheduler.py", line 2[359](https://github.com/dask/distributed/actions/runs/3494272750/jobs/5849912273#step:18:360), in transition_processing_memory

self._exit_processing_common(ts, recommendations)

File "D:\a\distributed\distributed\distributed\scheduler.py", line 3201, in _exit_processing_common

for qts in self._next_queued_tasks_for_worker(ws):

File "D:\a\distributed\distributed\distributed\scheduler.py", line 3218, in _next_queued_tasks_for_worker

assert qts.state == "queued", qts.state

AssertionError: forgotten

2022-11-18 05:04:47,523 - distributed.scheduler - INFO - Remove worker

2022-11-18 05:04:47,523 - distributed.core - INFO - Connection to tcp://127.0.0.1:52859 has been closed.

2022-11-18 05:04:47,523 - distributed.core - INFO - Removing comms to tcp://127.0.0.1:52864

2022-11-18 05:04:47,523 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:52864. Reason: worker-handle-scheduler-connection-broken

2022-11-18 05:04:47,820 - distributed.core - ERROR - forgotten
```
https://github.com/dask/distributed/actions/runs/3494272750/jobs/5849912273#step:18:352

This is weird. Maybe indicates an actual bug? I'm not sure how a task in the queue could be in state `forgotten`. `queued->released` removes the task from `scheduler.queue`. So any direct `queued->forgotten` transition would go through there first.

But given that the whole process dies, perhaps this is a red herring and something much deeper is broken, and this transition happens to be one manifestation of this thing breaking.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.