dotnet / dotnet/sdk

Test results in AzDO report unqualified method names after MSTest migration

Open
#55,123 6 comments 0 reactions 1 assignee Claimed by @Evangelink View on GitHub
Area-Infrastructure
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

# Test results in AzDO report unqualified method names after MSTest migration

## Summary

After the SDK repo's migration from xUnit to MSTest (MTP), test results published to Azure DevOps report only the bare method name (e.g., `GetMemory_DefaultCtor`) instead of the fully-qualified name (e.g., `Microsoft.DotNet.HotReload.UnitTests.ArrayBufferWriterTests_Byte.GetMemory_DefaultCtor`). This makes test results significantly harder to interpret and blocks reliable test history lookup.

## Problem

When viewing test results in AzDO for SDK PR validation builds, the `automatedTestName` field contains only the short method name without namespace or class context. For example:

| xUnit behavior | MSTest/MTP behavior |
| --- | --- |
| `Microsoft.DotNet.Cli.Utils.Tests.GivenAProjectDependencyCommandResolver.ItFindsToolsLocatedInTheNuGetPackageFolderAsTool` | `ItFindsToolsLocatedInTheNuGetPackageFolderAsTool` |
| `Microsoft.DotNet.HotReload.UnitTests.ArrayBufferWriterTests_Byte.GetMemory_DefaultCtor` | `GetMemory_DefaultCtor` |

## Impact

### 1. Difficult to identify test failures

When a test fails, the class name provides essential context for knowing the area the failure occurs in and locating the source. A failure reported as `ItCanBuildTemplates` could be in any of dozens of test classes. Engineers must dig into work item logs to find the actual source.

### 2. Ambiguous test names

Multiple test classes can (and do) have methods with the same name. For example, `CountEqualsZero_FixedAsync` appears in multiple NetAnalyzers test classes. In AzDO, these show up as duplicate entries with no way to distinguish which class they belong to without digging into the logs.

### 3. Breaks historical test duration lookup

We are investigating adopting Roslyn's time-based Helix scheduling approach (bin-packing test methods into work items targeting a time budget using historical execution data). This requires looking up per-test durations from prior builds. With unqualified names:

- Duplicate method names across classes make duration data unreliable

## Ideal Experience

Ideally, the AzDO Tests tab would present results in an expandable hierarchy:

```
▶ Microsoft.DotNet.HotReload.UnitTests.ArrayBufferWriterTests_Byte (21 tests, 2 failed)
✓ ArrayBufferWriter_Ctor
✓ Advance
✗ GetMemory_DefaultCtor
✗ GetMemory_ExceedMaximumBufferSize_WithSmallStartingSize
...
▶ Microsoft.DotNet.Cli.Utils.Tests.GivenAProjectDependencyCommandResolver (8 tests)
✓ ItFindsToolsLocatedInTheNuGetPackageFolderAsTool
...
```

Class names grouped hierarchically would provide immediate context about which area of the codebase is affected, while still allowing drill-down into individual method results. Even if a full tree view isn't supported by AzDO's UI, simply having `automatedTestName` contain the FQN would at least allow grouping/filtering by class name prefix.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.