dotnet / dotnet/sdk

macOS x64 public CI tests consistently failing with timeouts and hangs

Open
#54,988 0 comments 0 reactions 0 assignees View on GitHub
Area-Infrastructure untriaged
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

## Summary

The macOS x64 test leg (`osx.15.amd64.open`) in the `dotnet-sdk-public-ci` pipeline is consistently failing. These failures are not caught in PR validation because macOS x64 does not run there — they only manifest in the public CI builds against `main`.

## Failure Pattern

Out of 215 work items, 15 are failing in a recent build. The failures fall into two categories:

### 1. Timeouts (exit code -3) — 13 work items

Tests hang until the 1-hour Helix work item timeout is reached, then get killed. Example: `dotnet.Tests.dll.5` hangs indefinitely on `ItRecognizesRelativePathsForASolutionFromSubFolder`.

Affected test assemblies:
- `dotnet.Tests.dll` (shards 5, 9, 16, 22)
- `Microsoft.NET.Build.Tests.dll` (shards 2, 6, 7, 15)
- `Microsoft.NET.Sdk.BlazorWebAssembly.Tests.dll` (shard 1)
- `Microsoft.NET.Sdk.Razor.Tests.dll` (shard 2)
- `Microsoft.NET.Sdk.StaticWebAssets.Tests.dll` (shards 14, 15)
- `dotnet-watch.Tests.dll` (shard 7)

### 2. Test host crashes (exit code 1) — 2 work items

Test host becomes unresponsive (inactivity timeout of 48 minutes triggers hang dump collection), then the process is aborted.

- `Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll` (shards 7, 22)

The crash in shard 22 occurred while running: `Microsoft.NetCore.Analyzers.Usage.UnitTests.PreferGenericOverloadsTests.MultipleTypeArgumentsWithOtherGenericArgument_OffersFixer_CS`

## Example Build

- **Build**: [dotnet-sdk-public-ci #20260625.35](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1481771)
- **Helix Job**: `91c533ba-9018-4d10-b9ba-92d6fc23a862` ([details](https://helix.dot.net/api/jobs/91c533ba-9018-4d10-b9ba-92d6fc23a862/details))
- **Queue**: `osx.15.amd64.open`
- **Branch**: `refs/heads/main`
- **Commit**: 42b15a448cc1f5274d6287ebe31a84d572e18b8a

## Impact

These failures have been happening consistently for some time. Since macOS x64 is not part of PR validation, regressions land undetected and the public CI build remains persistently red for this leg.

## Notes

- The timeouts span many different test assemblies, suggesting a systemic issue on macOS x64 rather than individual test bugs.
- All timeout failures occur on different machines (dci-mac-build-*), ruling out a single bad machine.
- The 1-hour timeout is being hit across the board, indicating tests are hanging rather than being slow.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with public CI build dotnet-sdk-public-ci #20260625.35 and Helix job 91c533ba-9018-4d10-b9ba-92d6fc23a862, comparing the osx.15.amd64.open work items and hang dumps. Inspect the listed dotnet.Tests.dll, Microsoft.NET.Build.Tests.dll, and Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll failures, including PreferGenericOverloadsTests.MultipleTypeArgumentsWithOtherGenericArgument_OffersFixer_CS. Done means identifying and fixing the systemic cause so this macOS x64 leg completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
ci-cd, operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.