dotnet / dotnet/source-build

UX when there is failing PoisonTests in CI could be improved.

Open
#3,910 2 comments 0 reactions 0 assignees View on GitHub
area-testing
Dominant language
No language data
Stars
287
Forks
145
Avg merge
1d 22h
Merged PRs (30d)
10

Description

The following information is all that appears when there is a failing poison test in CI

```
Error message
Assert.Null() Failure: Value is not null
Expected: null
Actual: "\nExpected file '/vmr/test/Microsoft.DotNet.Source"···

Stack trace
[at Microsoft.DotNet.SourceBuild.SmokeTests.BaselineHelper.CompareFiles(String expectedFilePath, String actualFilePath, ITestOutputHelper outputHelper, Boolean warnOnDiffs) in /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/BaselineHelper.cs:line 76](https://dev.azure.com/dnceng/internal/_git/ddda0dd6-918f-48ba-ba8f-730d9d8c9320?path=%2Fvmr%2Ftest%2FMicrosoft.DotNet.SourceBuild.SmokeTests%2FBaselineHelper.cs&version=GBmain&_a=contents&line=76&lineEnd=77&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Microsoft.DotNet.SourceBuild.SmokeTests.BaselineHelper.CompareBaselineContents(String baselineFileName, String actualContents, ITestOutputHelper outputHelper, Boolean warnOnDiffs, String baselineSubDir) in /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/BaselineHelper.cs:line 49](https://dev.azure.com/dnceng/internal/_git/ddda0dd6-918f-48ba-ba8f-730d9d8c9320?path=%2Fvmr%2Ftest%2FMicrosoft.DotNet.SourceBuild.SmokeTests%2FBaselineHelper.cs&version=GBmain&_a=contents&line=49&lineEnd=50&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Microsoft.DotNet.SourceBuild.SmokeTests.PoisonTests.VerifyUsage() in /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/PoisonTests.cs:line 30](https://dev.azure.com/dnceng/internal/_git/ddda0dd6-918f-48ba-ba8f-730d9d8c9320?path=%2Fvmr%2Ftest%2FMicrosoft.DotNet.SourceBuild.SmokeTests%2FPoisonTests.cs&version=GBmain&_a=contents&line=30&lineEnd=31&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
```

Furthermore the attached console output only includes the following:

```
Executing: git diff --no-index /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net9.0/assets/baselines/PoisonUsage.txt /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net9.0/logs/UpdatedPoisonUsage.txt
```

It requires intimate knowledge of the pipeline to know that the new UpdatedPoisonUsage.txt file is captured as a build artifact.

One way to improve the UX would be to include the diff in the standard output.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.