dask / dask/distributed

Update CI stability

Open
#6,625 6 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

With a couple of recent merges, I triggered yesterday another "CI stress test" that runs our suite a couple of times in a row (this time 10)

see https://github.com/fjetter/distributed/tree/stress_ci
which is based on https://github.com/dask/distributed/commit/dc019ed2398411549cdf738be4327a601ec7dfca
with https://github.com/fjetter/distributed/commit/68689f0c9c41f0b1faf91888eea8d0be7745338a on top

The results of this test run can be seen https://github.com/fjetter/distributed/runs/7029246894?check_suite_focus=true

## Summary

We had overall 80 total jobs spread on the different OSs and python versions of which 32 failed.

| OS | success | failure | total | success rate |
| --- | --- | --- | --- | --- |
| Ubuntu | 28 | 2 | 30 | 93% |
| Windows | 13 | 17 | 30 | 43% |
| OSX | 7 | 13 | 20 | 35% |

If we look at an entire test run, i.e. a full test matrix for a given run number, **not a single job would've been successful.**

Looking at the kinds of test failures, we will see that three jobs on windows failed due to a [GH actions test timeout of 120s](https://github.com/fjetter/distributed/runs/7029246894?check_suite_focus=true#logs). And two test runs where [cancelled](https://github.com/fjetter/distributed/runs/7029246894?check_suite_focus=true#step:10:2850) by github without further information, also on windows.
The timing out test runs do not have anything obvious in common. In fact, one of the three timed out tests appears to have finished running the pytest suite but still timed out.

A modified test report available here https://gistpreview.github.io/?ecc2cdddf651df9ee0c7966e210c9093

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.