dask / dask/distributed

Automatically open issues for failing tests on `main`

Open
#6,969 3 comments 0 reactions 0 assignees View on GitHub
discussion enhancement tests
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

With recent work towards cleaning up CI, we've gotten a lot closer to all-green builds. After a couple more tests are fixed, hopefully the [short test report](https://dask.github.io/distributed/test_short_report.html) will be empty.

In order to _keep_ it empty, I'd like to add automation to help enforce a cultural practice of not ignoring flaky tests, and catching them quickly. I'd be happy to discuss other ways of doing this, but here's one simple thing that comes to mind.

Can we add a GitHub action that:
- opens an issue every time a new test fails on main
- comments on the existing issue when it fails again with a link to the new failure
- links to the first PR after which it failed
- tags the author of that PR, and core devs

Obviously, sometimes the author-tagging will be wrong (especially at the beginning, when we still have plenty of pre-existing flaky tests that may pop up randomly). Maybe we don't actually want it at first. But to me, it's initially more about a way of signaling a new attitude towards flaky tests, and increasing visibility. (Statistically, most PR authors are probably also core devs.)

Also, I opened this here, but it could probably happen on dask/dask too. @jrbourbeau want to keep this issue here, or move it to dask/community?

cc @hendrikmakait @martindurant @quasiben @fjetter @ian-r-rose @jsignell @crusaderky

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.