dotnet / dotnet/android

[ci] Test retries don't clean between runs

Open
#8,687 1 comment 0 reactions 0 assignees View on GitHub
Area: Unit Tests bug flaky-ci
Dominant language
C#
Stars
2.1k
Forks
579
Avg merge
1d 21h
Merged PRs (30d)
257

Description

For the tests that use Azure Pipelines' built-in `retryCountOnTaskFailure` (like `Mono.Android.NET-Tests`), the test suite is re-run immediately after the failure. We cannot inject any extra steps before the retry. This means that the second run is an incremental build because the `obj`/`bin` directories are not cleaned. Additionally, the logs from the first run are not captured.

This could lead to passing of tests that fail on initial builds and only succeed on incremental builds.

Given the inflexibility of `retryCountOnTaskFailure`, we likely need to turn it off completely.

We should also examine our usage of `dotnet-test-slicer` to see if it suffers from the same issue. At least with `dotnet-test-slicer` we can run steps between the initial run and the retry, so we can add manual cleans if needed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.