Improve awareness of frequently flaking tests
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**Problem:**
We have tests that are failing rather frequently. The [test report](https://dask.org/distributed/test_report.html) is a good resource to get an overview, but you have to actively take a look at it to identify those frequently failing tests. At the same time, #6452 has demonstrated that these flakes can point toward general issues that affect users. Had we taken a look earlier, we would have caught #6494 earlier as well.

**Question:**
How can we improve our awareness of and response time to tests that start flaking frequently?
**Possible Solution:**
One possible solution would be to implement a bot that files a new ticket for every test that starts flaking at least x times in the last y CI runs on `main`. The extreme being a ticket for _any_ failing test on `main`.
There also exist (paid) tools that allow tracking flaking tests in more detail.
cc @fjetter
Contributor guide
Assessment
This issue has not been assessed yet.