Update CI stability
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
With a couple of recent merges, I triggered yesterday another "CI stress test" that runs our suite a couple of times in a row (this time 10)
see https://github.com/fjetter/distributed/tree/stress_ci
which is based on https://github.com/dask/distributed/commit/dc019ed2398411549cdf738be4327a601ec7dfca
with https://github.com/fjetter/distributed/commit/68689f0c9c41f0b1faf91888eea8d0be7745338a on top
The results of this test run can be seen https://github.com/fjetter/distributed/runs/7029246894?check_suite_focus=true
## Summary
We had overall 80 total jobs spread on the different OSs and python versions of which 32 failed.
| OS | success | failure | total | success rate |
| --- | --- | --- | --- | --- |
| Ubuntu | 28 | 2 | 30 | 93% |
| Windows | 13 | 17 | 30 | 43% |
| OSX | 7 | 13 | 20 | 35% |
If we look at an entire test run, i.e. a full test matrix for a given run number, **not a single job would've been successful.**
Looking at the kinds of test failures, we will see that three jobs on windows failed due to a [GH actions test timeout of 120s](https://github.com/fjetter/distributed/runs/7029246894?check_suite_focus=true#logs). And two test runs where [cancelled](https://github.com/fjetter/distributed/runs/7029246894?check_suite_focus=true#step:10:2850) by github without further information, also on windows.
The timing out test runs do not have anything obvious in common. In fact, one of the three timed out tests appears to have finished running the pytest suite but still timed out.
A modified test report available here https://gistpreview.github.io/?ecc2cdddf651df9ee0c7966e210c9093
Contributor guide
Assessment
This issue has not been assessed yet.