[ci-scan] Hang: maccatalyst-arm64 AllSubsets_CoreCLR_RuntimeTests time out (Test run never launched) on osx.26.arm64
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
## Build Information
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1584171
Build error leg or test failing: maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests - osx.26.arm64.open — CoreCLR runtime test work items (Exceptions, Interop, jit64_3, Methodical_*, threading_group2, Regression_PdbOnly_r_1)
KBE authoring guidance (ci-failure-scan)
- `Error Details` is for readers. Paste the full exception, stack trace, or build error excerpt so the failure is understandable without opening the raw log.
- Build Analysis parses only the single JSON block under `Error Message`.
- `ErrorMessage` is a case-sensitive ordinal `String.Contains` substring copied verbatim from the failing log.
- Set `BuildRetry` to `true` only for a clear infrastructure retry case. `ExcludeConsoleLog` disables Helix console-log scanning.
## Error Details
On the `runtime-extra-platforms` pipeline (definition 154), the `maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests - osx.26.arm64.open` leg fails repeatedly: multiple CoreCLR runtime-test work items never launch the test application and are killed after the 3-hour (10800 second) work-item timeout. The harness reports that the process could never be attached and that the test run never launched, indicating an app-launch/attach failure on the `osx.26.arm64` MacCatalyst queue rather than a test assertion.
```
[03:38:13] fail: Run timed out after 10800 seconds
[03:38:13] dbug: Pids to kill: 25661
[03:38:13] dbug: Running lldb diagnostics for pid 25661
[03:38:14] dbug: (lldb) process attach --pid 25661
[03:38:14] dbug: error: attach failed: no such process
[03:38:14] dbug: 25661 Execution timed out after 10800 seconds and the process was killed.
[03:38:14] dbug: Test run timed out after 180 minute(s).
[03:38:14] dbug: Could not find pid in mtouch output.
[03:38:14] dbug: Test run never launched
```
Observed across work items in build 1584171 (source) and reproduced in follow-up build 1584749 (`Exceptions`, `Interop`, `Regression_PdbOnly_r_1`, `threading_group2`), so the failure is stable across builds and work items on this leg.
## Error Message
```json
{
"ErrorMessage": [
"Run timed out after 10800 seconds",
"Test run never launched"
],
"ErrorPattern": "",
"BuildRetry": false,
"ExcludeConsoleLog": false
}
```
Agentic workflow metadata (ci-failure-scan)
Workflow artifact: ci-failure-scan
Artifact kind: kbe-verification
Verified match count: both lines present in failure.log for each failed work item; observed across 8 maccatalyst-arm64 RuntimeTests work items in source build 1584171 and reproduced in follow-up 1584749
---
> [!NOTE]
> This issue was filed automatically by the `ci-failure-scan` agentic workflow (AI/Copilot-generated). Mitigation is handled separately by the `ci-failure-fix` workflow.
> Generated by [CI Outer-Loop Failure Scanner](https://github.com/dotnet/runtime/actions/runs/34072178832) · opus48 · 814.4 AIC · ⌖ 20 AIC · ⊞ 20.7K · [◷](https://github.com/search?q=repo%3Adotnet%2Fruntime+is%3Aissue+%22gh-aw-workflow-call-id%3A+dotnet%2Fruntime%2Fci-failure-scan%22&type=issues)
### Report
|Build|Repository|Test|Pull Request|
|---|---|---|---|
|[1596986](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1596986)|dotnet/runtime|[JIT_others.WorkItemExecution](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1596986&view=ms.vss-test-web.build-test-results-tab&runId=44087194&resultId=103069)||
|[1594919](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1594919)|dotnet/runtime|[Methodical_d1.WorkItemExecution](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1594919&view=ms.vss-test-web.build-test-results-tab&runId=44031390&resultId=102852)||
|[1594486](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1594486)|dotnet/runtime|[Loader.WorkItemExecution](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1594486&view=ms.vss-test-web.build-test-results-tab&runId=44015422&resultId=102948)||
|[1593920](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1593920)|dotnet/runtime|[JIT.CodeGenBringUpTests_r.WorkItemExecution](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1593920&view=ms.vss-test-web.build-test-results-tab&runId=43996066&resultId=103189)||
|[1592042](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1592042)|dotnet/runtime|[threading_group2.WorkItemExecution](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1592042&view=ms.vss-test-web.build-test-results-tab&runId=43947258&resultId=103773)||
|[1590326](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1590326)|dotnet/runtime|[jit64_3.WorkItemExecution](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1590326&view=ms.vss-test-web.build-test-results-tab&runId=43896318&resultId=103353)||
#### Summary
|24-Hour Hit Count|7-Day Hit Count|1-Month Count|
|---|---|---|
|1|6|6|
Contributor guide
Research direction
Start with the failure logs for build 1584171 and follow-up build 1584749, focusing on the maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests leg and its work items. Compare the timeout, lldb attach, and mtouch output to identify where app launch or attachment stops. Done means the affected work items launch and complete on osx.26.arm64 without the 10800-second timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd, mobile-dev, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100