dotnet / dotnet/aspnetcore

Fail AzDO Pipeline if number of passed tests fall below a certain threashold.

Open
#24,687 5 comments 1 reaction 0 assignees View on GitHub
affected-few area-infrastructure enhancement severity-nice-to-have
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 6h
Merged PRs (30d)
290

Description

### Is your feature request related to a problem? Please describe.

This week, the number of passed tests as part of our daily CI builds and PR checks this week decreased from [~85,000](https://dev.azure.com/dnceng/public/_build/results?buildId=758545&view=ms.vss-test-web.build-test-results-tab) to [~20,000](https://dev.azure.com/dnceng/public/_build/results?buildId=760441&view=ms.vss-test-web.build-test-results-tab). Despite this, the PR checks passed an the build stayed green.

![image](https://user-images.githubusercontent.com/54385/89686643-93e18f00-d8b3-11ea-860e-1c58bd88999a.png)

![image](https://user-images.githubusercontent.com/54385/89686800-dc00b180-d8b3-11ea-8ab5-a332939a30e3.png)

The tests stopped running because of an issue with xunit/arcade (see https://github.com/dotnet/arcade/pull/5908) that caused `dotnet test` runs to fail silently, but I think automated checks should have caught the dramatic reduction in the number of passed tests.

### Describe the solution you'd like

We should set some sort of minimum threshold on the number of passed tests for a CI build/PR check pass. If we ever intentionally reduce the number of tests we run intentionally, we can manually adjust this threshold.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.