dotnet / dotnet/xharness

`OuterLoop`, and `SkipOnPlatform` attributes conflict, and don't work correctly when used together

Open
#908 4 comments 0 reactions 1 assignee Claimed by @akoeplinger View on GitHub
test-runners
Dominant language
C#
Stars
189
Forks
67
Avg merge
1d 12h
Merged PRs (30d)
29

Description

Consider
https://github.com/dotnet/runtime/blob/978df67ced885aeca5f7e75379451bc1a57cc219/src/libraries/System.Net.WebSockets/tests/WebSocketCreateTest.cs#L40-L45

```csharp
[OuterLoop("Uses external servers")]
[Theory]
[MemberData(nameof(EchoServers))]
[SkipOnPlatform(TestPlatforms.Browser, "System.Net.Sockets is not supported on this platform.")]
[ActiveIssue("https://github.com/dotnet/runtime/issues/34690", TestPlatforms.Windows, TargetFrameworkMonikers.Netcoreapp, TestRuntimes.Mono)]
public async Task WebSocketProtocol_CreateFromConnectedStream_CanSendReceiveData(Uri echoUri)
```

This has both `OuterLoop`, and `SkipOnPlatform` attributes. This should run when using `category=outerloop`, but get skipped if it is running on the browser. Instead, the tests run anyway, and fail because the operations are not supported on browser.

I can reproduce this with `./dotnet.sh build /bl src/libraries/System.Net.WebSockets/tests/System.Net.WebSockets.Tests.csproj /p:Configuration=Release /p:TargetOS=Browser /p:TargetArchitecture=wasm /t:Test -p:Scenario=WasmTestOnBrowser -p:TestScope=outerloop -p:XUnitClassName=System.Net.WebSockets.Tests.WebSocketTests`

this is the xharness command line:
` wasm test-browser --app=. --output-directory=/workspaces/runtime/artifacts/bin/System.Net.WebSockets.Tests/Release/net7.0/browser-wasm/AppBundle/xharness-output
--browser-arg=--no-sandbox -s dotnet.js.symbols --symbol-patterns wasm-symbol-patterns.txt --browser-arg=--no-sandbox -- --run WasmTestRunner.dll System.Net.WebSockets.Tests.dll -class System.Net.WebSockets.Tests.WebSocketTests -trait category=OuterLoop -notrait category=failing`.

I don't think this is wasm, or `SkipOnPlatform` specific.

cc @akoeplinger @premun

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.