dotnet / dotnet/arcade

Build Analysis misclassifies known Helix test failures after telemetry category removal

Open
#17,340 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
729
Forks
397
Avg merge
3d 15m
Merged PRs (30d)
149

Description

# Build Analysis misclassifies known Helix test failures after telemetry category removal

## Description

Build Analysis can remain red when every failing Helix test is matched to a known issue. The Helix SDK reports each failed test through `CheckAzurePipelinesTestResults` using `Log.LogError(FailureCategory.Test, ...)`, but these errors now reach the Azure DevOps timeline as plain MSBuild errors:

```text
Microsoft.DotNet.Helix.Sdk/.../AzurePipelines.MultiQueue.targets(44,5): error : Test has failed.
```

Build Analysis correctly matches the corresponding test results to known issues, but also treats the unclassified timeline messages as independent build failures. The check therefore remains red.

This appears related to:

- dotnet/arcade#16814, which removed `Microsoft.DotNet.ArcadeLogging` and the `(NETCORE_ENGINEERING_TELEMETRY=)` decoration.
- dotnet/arcade#17110, which restored a minimal logger that forwards MSBuild errors and warnings to the Azure DevOps timeline, but did not restore failure-category propagation.

The Helix task still supplies `FailureCategory.Test`; that classification is no longer represented in the timeline message consumed by Build Analysis.

## Example

- PR: https://github.com/dotnet/runtime/pull/132290
- Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1552976
- Helix SDK: `11.0.0-beta.26381.1` from Arcade commit `93eebf1a31a5eaafd44326f1a81ca107913e098c`

The build had two failed tests:

1. `Loader/ContextualReflection/ContextualReflection/ContextualReflection.cmd`, matched to dotnet/runtime#131925.
2. `Loader\classloader\StaticVirtualMethods\Regression\GitHub_130545\GitHub_130545.dll`, matched to dotnet/runtime#132030.

Build Analysis displayed both under **Known test errors**, but also displayed the `AzurePipelines.MultiQueue.targets(44,5)` messages under **Build Failures**. Those duplicate build failures kept the check red.

## Expected behavior

Once every failed test is matched to a known issue, the corresponding Helix task-level errors should not independently fail Build Analysis.

## Actual behavior

The known test failures are matched, but their plain MSBuild timeline errors remain unmatched and cause Build Analysis to fail.

## Possible fixes

Either:

- preserve `FailureCategory.Test` in a form Build Analysis can consume when forwarding MSBuild events to the Azure DevOps timeline, or
- update Build Analysis to recognize and de-duplicate `CheckAzurePipelinesTestResults` errors against the corresponding failed test results.

Contributor guide

Open the contributing guide

Research direction

Start by tracing CheckAzurePipelinesTestResults and the AzurePipelines.MultiQueue.targets error at line 44, then inspect how FailureCategory.Test reaches the Azure DevOps timeline and how Build Analysis consumes it. Done means known Helix test failures no longer produce separate unmatched Build Failures when every failed test is matched to a known issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
build-system, ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.