[CI] Sonoma Clean bot hangs in simctl delete all for 60 minutes
- Dominant language
- C#
- Stars
- 2.9k
- Forks
- 576
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 123
Description
## Summary
The Sonoma `Clean bot` task has hung in `simctl delete all` until Azure DevOps terminates the task at its 60-minute timeout in two distinct monitored PRs.
- Stable signature: `X64 Mac Sonoma clean-bot.sh hangs in simctl delete all until the 60-minute Azure task timeout`
- Classification: CI infrastructure failure
- Distinct monitored PRs: 2
- Confirmed occurrences: 2
- First observed timeout: 2026-09-01T18:48:27Z
- Latest observed timeout: 2026-09-02T03:55:54Z
- Worker in both occurrences: `XAMBOT-1024.Sonoma`
## Confirmed occurrences
### #26475
- Head: `2ddfa5661d7ae0592ef92481927aa61f28ba8df5`
- Pipeline: `xamarin-macios-sim-pr-tests`
- [Build 15169876, attempt 1, X64 Mac Sonoma / Clean bot](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15169876&view=logs&j=623bc498-120b-5ff2-984a-5b073efffe1b&t=a1e6c3cd-490b-50ca-e13b-051fbafca435)
- Task interval: 2026-09-01T17:48:17Z–18:48:27Z
- Minimal signature:
- `+ /Applications/Xcode.app/Contents/Developer/usr/bin/simctl delete all`
- `The task has timed out.`
- Azure build result: `partiallySucceeded`.
- Retry outcome: no cleanup retry was submitted. A separate permitted retry addressed the unrelated `FinalizationRaceCondition` test failure and passed; the cleanup timeout remained as a nonblocking warning.
### #26484
- Head: `fd099f147cffffbce1c6a10b5861f4172fd8117b`
- Pipeline: `xamarin-macios-sim-pr-tests`
- [Build 15178389, attempt 1, X64 Mac Sonoma / Clean bot](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15178389&view=logs&j=623bc498-120b-5ff2-984a-5b073efffe1b&t=a1e6c3cd-490b-50ca-e13b-051fbafca435)
- Task interval: 2026-09-02T02:55:44Z–03:55:54Z
- Minimal signature:
- `+ /Applications/Xcode.app/Contents/Developer/usr/bin/simctl delete all`
- `The task has timed out.`
- Azure build result: `partiallySucceeded`; all GitHub checks are green.
- Retry outcome: none. The timeout was nonblocking and no failed test or build job remained to rerun.
## Reproduction pattern and impact
Both occurrences used the same job, task identifier, command, operating-system lane, and worker. The command produced no completion output for about 60 minutes and occupied the agent until Azure timed it out. Tests were not invalidated, but the task delayed each pipeline and left the Azure build partially successful.
## Investigation performed
The Azure build metadata, timelines, task records, and `Clean bot` logs were compared directly. The exact command, task timeout text, task duration, and worker match across both PRs. Searches across all open and closed `dotnet/macios` issues used the exact command, task name, timeout text, worker, script name, and simulator-cleanup terms. The only candidates were #6740, an unrelated Xcode UI synchronization report, and #26385, an unrelated checkout/credential failure.
A binlog, TestSummary, HtmlReport, or crash report is not applicable because this is a shell-based post-test simulator cleanup hang rather than a build, test assertion, or process crash.
## Current diagnosis and workaround
This is a worker-specific CoreSimulator cleanup infrastructure failure. No source workaround is known. The affected worker's CoreSimulator state should be inspected or reset, and the cleanup path should capture bounded diagnostics before timing out so future hangs identify the blocking simulator service or process.
Azure logs are retention-limited.
Contributor guide
No contributing guide indexed for this repository
Research direction
Review clean-bot.sh and the Azure Clean bot logs, starting with the simctl delete all invocation on XAMBOT-1024.Sonoma. Inspect the affected worker's CoreSimulator state and determine how the cleanup path can capture bounded diagnostics before timeout. Done means the cleanup no longer occupies the agent for 60 minutes, or the failure provides actionable diagnostics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, macos, shell
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100