Automatically open issues for failing tests on `main`
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
With recent work towards cleaning up CI, we've gotten a lot closer to all-green builds. After a couple more tests are fixed, hopefully the [short test report](https://dask.github.io/distributed/test_short_report.html) will be empty.
In order to _keep_ it empty, I'd like to add automation to help enforce a cultural practice of not ignoring flaky tests, and catching them quickly. I'd be happy to discuss other ways of doing this, but here's one simple thing that comes to mind.
Can we add a GitHub action that:
- opens an issue every time a new test fails on main
- comments on the existing issue when it fails again with a link to the new failure
- links to the first PR after which it failed
- tags the author of that PR, and core devs
Obviously, sometimes the author-tagging will be wrong (especially at the beginning, when we still have plenty of pre-existing flaky tests that may pop up randomly). Maybe we don't actually want it at first. But to me, it's initially more about a way of signaling a new attitude towards flaky tests, and increasing visibility. (Statistically, most PR authors are probably also core devs.)
Also, I opened this here, but it could probably happen on dask/dask too. @jrbourbeau want to keep this issue here, or move it to dask/community?
cc @hendrikmakait @martindurant @quasiben @fjetter @ian-r-rose @jsignell @crusaderky
Contributor guide
Assessment
This issue has not been assessed yet.