dotnet / dotnet/dotnet-docker

Skipped unit tests should not show as "passed"

Open
#6,167 1 comment 0 reactions 0 assignees View on GitHub
area-infrastructure up-for-grabs
Dominant language
Dockerfile
Stars
4.9k
Forks
2k
Avg merge
1d 14h
Merged PRs (30d)
26

Description

We have lots of instances of skipping tests in this manner:

https://github.com/dotnet/dotnet-docker/blob/ccbb4f2e87307ef063c576f638ee2416d7a5d57e/tests/Microsoft.DotNet.Docker.Tests/RuntimeDepsImageTests.cs#L33-L38

However, returning early from a test will count as a "passed" test in Xunit, even if we didn't assert anything. This can lead to a false sense of confidence in test outcomes, especially for tests which are intended to be skipped temporarily.

We should find a way to make these tests show as "skipped" instead, like we do with Theories that have no data due to test filtering.

Xunit V3 (which is in preview) supports dynamic skipping of tests natively, but not Xunit V2. There's opportunity to use a library like [Xunit.SkippableFact](https://www.nuget.org/packages/Xunit.SkippableFact), which is used by several other Microsoft projects: [Xunit.SkippableFact > used by ...](https://www.nuget.org/packages/Xunit.SkippableFact#usedby-body-tab).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.