dotnet / dotnet/sdk

FileUpdateTests.TestCommand hangs in CI due to thread pool starvation from leaked PollingDirectoryWatcher threads

Open
#54,176 0 comments 0 reactions 0 assignees View on GitHub
Area-Watch untriaged
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

## Description

`FileUpdateTests.TestCommand` hangs in CI (Helix) waiting for `Passed!` output from the child `dotnet watch test` process that never arrives.

**AzDO Build**: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1405748
**Helix Job**: `bca8b3a7-8159-4afe-86d6-dfce52fb7cda`, work item `dotnet-watch.Tests.dll.4`

## Dump Analysis

The test runner process dump (`dotnet-watch.Tests_7656`) shows:

1. **Test stuck at**: `FileUpdateTests+d__3` state 2, awaiting `AwaitableProcess.GetOutputLineAsync` -> waiting for a line containing `Passed!` that never comes from the child `dotnet watch test` process.

2. **695 `PollingDirectoryWatcher.PollingLoop()` threads** leaked in the test runner process from concurrent in-process watch tests, likely contributing to thread pool starvation.

3. The child `dotnet watch test` process either failed to detect the file change, failed to rebuild, or failed to produce `Passed!` test output.

## The Test

The test (`test/dotnet-watch.Tests/Watch/FileUpdateTests.cs`, line 55-96) validates the `dotnet watch test` scenario from #52528:
1. Writes a failing xunit test
2. Starts `dotnet watch test`
3. Waits for `Failed!` and `WaitingForFileChangeBeforeRestarting`
4. Updates the test file to make it pass
5. Waits for `Passed!` <- **hangs here**

## Root Cause

The test involves multiple nested processes (test runner -> dotnet watch -> dotnet test -> testhost) and file system polling in CI. The combination of thread pool starvation (695 polling threads) and multi-process coordination makes this inherently flaky.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.