Build Analysis misclassifies known Helix test failures after telemetry category removal
- Dominant language
- C#
- Stars
- 729
- Forks
- 397
- Avg merge
- 3d 15m
- Merged PRs (30d)
- 149
Description
# Build Analysis misclassifies known Helix test failures after telemetry category removal
## Description
Build Analysis can remain red when every failing Helix test is matched to a known issue. The Helix SDK reports each failed test through `CheckAzurePipelinesTestResults` using `Log.LogError(FailureCategory.Test, ...)`, but these errors now reach the Azure DevOps timeline as plain MSBuild errors:
```text
Microsoft.DotNet.Helix.Sdk/.../AzurePipelines.MultiQueue.targets(44,5): error : Test has failed.
```
Build Analysis correctly matches the corresponding test results to known issues, but also treats the unclassified timeline messages as independent build failures. The check therefore remains red.
This appears related to:
- dotnet/arcade#16814, which removed `Microsoft.DotNet.ArcadeLogging` and the `(NETCORE_ENGINEERING_TELEMETRY=)` decoration.
- dotnet/arcade#17110, which restored a minimal logger that forwards MSBuild errors and warnings to the Azure DevOps timeline, but did not restore failure-category propagation.
The Helix task still supplies `FailureCategory.Test`; that classification is no longer represented in the timeline message consumed by Build Analysis.
## Example
- PR: https://github.com/dotnet/runtime/pull/132290
- Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1552976
- Helix SDK: `11.0.0-beta.26381.1` from Arcade commit `93eebf1a31a5eaafd44326f1a81ca107913e098c`
The build had two failed tests:
1. `Loader/ContextualReflection/ContextualReflection/ContextualReflection.cmd`, matched to dotnet/runtime#131925.
2. `Loader\classloader\StaticVirtualMethods\Regression\GitHub_130545\GitHub_130545.dll`, matched to dotnet/runtime#132030.
Build Analysis displayed both under **Known test errors**, but also displayed the `AzurePipelines.MultiQueue.targets(44,5)` messages under **Build Failures**. Those duplicate build failures kept the check red.
## Expected behavior
Once every failed test is matched to a known issue, the corresponding Helix task-level errors should not independently fail Build Analysis.
## Actual behavior
The known test failures are matched, but their plain MSBuild timeline errors remain unmatched and cause Build Analysis to fail.
## Possible fixes
Either:
- preserve `FailureCategory.Test` in a form Build Analysis can consume when forwarding MSBuild events to the Azure DevOps timeline, or
- update Build Analysis to recognize and de-duplicate `CheckAzurePipelinesTestResults` errors against the corresponding failed test results.
Contributor guide
Research direction
Start by tracing CheckAzurePipelinesTestResults and the AzurePipelines.MultiQueue.targets error at line 44, then inspect how FailureCategory.Test reaches the Azure DevOps timeline and how Build Analysis consumes it. Done means known Helix test failures no longer produce separate unmatched Build Failures when every failed test is matched to a known issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- build-system, ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100