dask / dask/distributed

Flakiness in test_shuffle.py

Open
#8,074 3 comments 0 reactions 0 assignees View on GitHub
flaky test tests
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Several tests in test_shuffle.py are very flaky.

If I change `.github/workflows/tests.yaml` as follows, to rerun the tests 20 times (ci1 + not ci1) per environment:
```
pytest distributed/shuffle/tests/test_shuffle.py --count=10 --runslow \
--leaks=...
```
I get the following failure rates:

| test | n. failures |
| --- | --- |
| distributed/shuffle/tests/test_shuffle.py::test_clean_after_close | 1 |
| distributed/shuffle/tests/test_shuffle.py::test_closed_input_only_worker_during_transfer | 1 |
| distributed/shuffle/tests/test_shuffle.py::test_closed_worker_during_transfer | 29 |
| distributed/shuffle/tests/test_shuffle.py::test_crashed_worker_during_transfer | 6 |
| distributed/shuffle/tests/test_shuffle.py::test_restarting_during_transfer_raises_killed_worker | 38 |

Additionally, `test_crashed_worker_during_transfer` deadlocks in a way that's irrecoverable on Windows, causing the whole test suite to be killed by
https://github.com/dask/distributed/blob/eb297b3e7f4d045f80be0a5b50b42a579adc37ee/pyproject.toml#L155-L162

logs: https://github.com/crusaderky/distributed/actions/runs/5761255813

CC @hendrikmakait

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.