macOS x64 public CI tests consistently failing with timeouts and hangs
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 1.3k
- PR merge metrics
- PR metrics pending
Description
## Summary
The macOS x64 test leg (`osx.15.amd64.open`) in the `dotnet-sdk-public-ci` pipeline is consistently failing. These failures are not caught in PR validation because macOS x64 does not run there — they only manifest in the public CI builds against `main`.
## Failure Pattern
Out of 215 work items, 15 are failing in a recent build. The failures fall into two categories:
### 1. Timeouts (exit code -3) — 13 work items
Tests hang until the 1-hour Helix work item timeout is reached, then get killed. Example: `dotnet.Tests.dll.5` hangs indefinitely on `ItRecognizesRelativePathsForASolutionFromSubFolder`.
Affected test assemblies:
- `dotnet.Tests.dll` (shards 5, 9, 16, 22)
- `Microsoft.NET.Build.Tests.dll` (shards 2, 6, 7, 15)
- `Microsoft.NET.Sdk.BlazorWebAssembly.Tests.dll` (shard 1)
- `Microsoft.NET.Sdk.Razor.Tests.dll` (shard 2)
- `Microsoft.NET.Sdk.StaticWebAssets.Tests.dll` (shards 14, 15)
- `dotnet-watch.Tests.dll` (shard 7)
### 2. Test host crashes (exit code 1) — 2 work items
Test host becomes unresponsive (inactivity timeout of 48 minutes triggers hang dump collection), then the process is aborted.
- `Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll` (shards 7, 22)
The crash in shard 22 occurred while running: `Microsoft.NetCore.Analyzers.Usage.UnitTests.PreferGenericOverloadsTests.MultipleTypeArgumentsWithOtherGenericArgument_OffersFixer_CS`
## Example Build
- **Build**: [dotnet-sdk-public-ci #20260625.35](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1481771)
- **Helix Job**: `91c533ba-9018-4d10-b9ba-92d6fc23a862` ([details](https://helix.dot.net/api/jobs/91c533ba-9018-4d10-b9ba-92d6fc23a862/details))
- **Queue**: `osx.15.amd64.open`
- **Branch**: `refs/heads/main`
- **Commit**: 42b15a448cc1f5274d6287ebe31a84d572e18b8a
## Impact
These failures have been happening consistently for some time. Since macOS x64 is not part of PR validation, regressions land undetected and the public CI build remains persistently red for this leg.
## Notes
- The timeouts span many different test assemblies, suggesting a systemic issue on macOS x64 rather than individual test bugs.
- All timeout failures occur on different machines (dci-mac-build-*), ruling out a single bad machine.
- The 1-hour timeout is being hit across the board, indicating tests are hanging rather than being slow.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with public CI build dotnet-sdk-public-ci #20260625.35 and Helix job 91c533ba-9018-4d10-b9ba-92d6fc23a862, comparing the osx.15.amd64.open work items and hang dumps. Inspect the listed dotnet.Tests.dll, Microsoft.NET.Build.Tests.dll, and Microsoft.CodeAnalysis.NetAnalyzers.UnitTests.dll failures, including PreferGenericOverloadsTests.MultipleTypeArgumentsWithOtherGenericArgument_OffersFixer_CS. Done means identifying and fixing the systemic cause so this macOS x64 leg completes successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- ci-cd, operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100