dask / dask/distributed

Improve awareness of frequently flaking tests

Open
#6,500 0 comments 0 reactions 0 assignees View on GitHub
discussion flaky test stability
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Problem:**
We have tests that are failing rather frequently. The [test report](https://dask.org/distributed/test_report.html) is a good resource to get an overview, but you have to actively take a look at it to identify those frequently failing tests. At the same time, #6452 has demonstrated that these flakes can point toward general issues that affect users. Had we taken a look earlier, we would have caught #6494 earlier as well.
![image](https://user-images.githubusercontent.com/2699097/171908050-ec48b6f1-b8de-4dc8-8b2a-f7479a9bf7f6.png)

**Question:**
How can we improve our awareness of and response time to tests that start flaking frequently?

**Possible Solution:**
One possible solution would be to implement a bot that files a new ticket for every test that starts flaking at least x times in the last y CI runs on `main`. The extreme being a ticket for _any_ failing test on `main`.

There also exist (paid) tools that allow tracking flaking tests in more detail.

cc @fjetter

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.