dotnet / dotnet/orleans

Flaky CI: MTP/TRX host fails when suite filters select zero tests

Closed
#10,754 9 comments 0 reactions 0 assignees View on GitHub
area-build area-testing bug Status: investigating test reliability test-issue
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
13h 56m
Merged PRs (30d)
351

Description

## Description

`Orleans.Analyzers.Tests` intermittently causes a filtered Microsoft.Testing.Platform CI job to fail when the active suite filter selects zero tests from that assembly. The suite continues and reports no failed tests, but the assembly exits unsuccessfully during test-host shutdown.

This has reproduced on unrelated pull requests and different operating systems/frameworks.

## Occurrences

### PR #10746

- Run/job: https://github.com/dotnet/orleans/actions/runs/32499721406/job/96826364553
- Environment: Functional, Windows, `net8.0`
- Head: `d1fe65475179923ba00ba6941fc077305151851a`
- PR scope: documentation only

```text
Orleans.Analyzers.Tests.dll (net8.0|x64) Zero tests ran (28s 485ms)
Exit code: 7
Standard output: Waiting 10 seconds for foreground threads to exit...
Error output: [FATAL ERROR] Foreground threads were left running, forcing process exit
```

The overall run reported 501 total tests, 482 succeeded, 19 skipped, 0 failed, and 1 test-application error.

### PR #10739

- Run: https://github.com/dotnet/orleans/actions/runs/32497565314
- Environment: SlowBVT, Ubuntu, `net10.0`
- PR scope: journaling recovery behavior and tests

```text
Orleans.Analyzers.Tests.dll (net10.0|x64) Zero tests ran (37s 377ms)
Exit code: 134
Unhandled exception. System.TimeoutException: The operation has timed out.
at Microsoft.Testing.Platform.Helpers.TaskExtensions.TimeoutAfterAsync(...)
at Microsoft.Testing.Extensions.TrxReport.Abstractions.TrxProcessLifetimeHandler...
```

## Expected behavior

A test application containing no tests which match the suite filter exits successfully and releases all MTP/TRX resources without delaying or failing the aggregate CI job.

## Notes

The failures occur in the analyzer test assembly under suite filters which select no analyzer tests. The different shutdown symptoms may share test-host or TRX lifetime cleanup behavior. The failed PR #10746 job is being rerun to confirm the intermittent nature.

Contributor guide

Open the contributing guide

Research direction

Start with the Orleans.Analyzers.Tests test application and reproduce a filtered run that selects zero tests on Microsoft.Testing.Platform with TRX enabled. Trace the test-host shutdown and TRX lifetime cleanup, comparing the Windows net8.0 and Ubuntu net10.0 failures described here. Done means a zero-test filtered application exits successfully, releases its resources, and does not fail or delay the aggregate CI job.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.