dask / dask/distributed

Group similar test failures in test dashboard?

Open
#6,778 0 comments 1 reaction 0 assignees View on GitHub
tests
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In https://github.com/dask/distributed/issues/6774, I manually went through the CI dashboard and identified taxonomies of test failures that were similar/the same across many tests. I have a feeling it would be valuable (and reasonable effort) to create a view that does this automatically, helping identify high-impact issues affecting CI.

Generally, there are probably 2 reasons for flaky tests:
1. An individual test is written in a way that's unreliable (too reliant on timing, actually causes a deadlock sometimes, etc.).
2. A bug in dask is causing something unrelated to the test to fail (timeout connecting to the cluster, asyncio error during cluster teardown, etc.). These tend to pop up in many unrelated tests. Because they can happen anywhere, they tend to blow up CI and the flaky test dashboard, and are probably responsible for the majority of failing tests.

I think we could identify #2 in a more automated way, just by creating another view on the test dashboard that groups failures by the failure message (like how `OSError: Timed out trying to connect to tcp://127.0.0.1:8786 after 5 s` shows up in 13 different tests). There would need to be some fuzziness to this (an exact string match wouldn't work). But that visibility might help us to identify, prioritize, and fix the problems faster. It's also possible that these systematic problems would be more likely to affect users?

cc @ian-r-rose

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.