getsentry / getsentry/sentry-cocoa

Intelligent flakiness mitigation

Open
#3,897 4 comments 0 reactions 0 assignees View on GitHub
Cocoa
Dominant language
Swift
Stars
1.1k
Forks
418
Avg merge
2d 3h
Merged PRs (30d)
106

Description

### Description

We've had a persistent problem with flaky tests for a long time.

One process we've tried using is to disable a flaky test with a PR, and open an issue to fix it. This has not eradicated the issues, and doesn't scale that well.

One automation we've attempted in the past is to allow up to a certain number of retries of any test invocation that is known to contain flakes. It was recently removed for our saucelabs runs in https://github.com/getsentry/sentry-cocoa/pull/3660 but still is in use for our UI test ASAN runs:https://github.com/getsentry/sentry-cocoa/blob/e0904ef6b8b510e9739820aa29379b074b26e237/.github/workflows/ui-tests.yml#L96

However, we have unit and UI tests that basically always contain at least one flake. This requires each PR author to check the test run, read what failed, decide if it's a real failure related to their changes, and use the web UI to initiate a retry of the failed tests, and then wait for that round to complete again. This process adds hours to each PR.

I propose automating that manual process the same way we've done it elsewhere, by using the retry strategy with https://github.com/getsentry/sentry-cocoa/pull/3881 and https://github.com/getsentry/sentry-cocoa/pull/3883.

Then, we should write some automation that will extract the name of any test that did not fail in every retry, where more than one test run was actually required, and report that in a PR comment so we can more easily perform our current flaky test process of disabling it and filing an issue to fix it, vs having to dive into the GHA logs for each one.

Even that could potentially be automated, since we can open github issues with their API, and skipping a test is just inserting a line of text into the .xcscheme file. We could also pipe the event to slack so we are aware of it happening there, too.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.