[CI] AcesShared agent hangs until job timeout and ignores cancellation
- Dominant language
- C#
- Stars
- 2.9k
- Forks
- 576
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 96
Description
## Failure signature
An AcesShared agent becomes unresponsive during an active task:
- The task produces no finalized Azure DevOps log.
- The job reaches the 180-minute hard limit.
- Azure reports that the agent did not respond to cancellation within five minutes.
- Post-job diagnostics and artifact publication cannot complete reliably.
Stable signature:
```text
AcesShared agent becomes unresponsive during an active task, produces no finalized task log, exceeds the 180-minute job limit, and ignores the 5-minute cancellation request
```
## Confirmed occurrences
### PR #26424
- Head: `b4abc54c0c077b0e1515c5b386a94b723f76f8d1`
- Build: [15164848, attempt 1, `T: framework` / `Install dotnet workloads`](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15164848&view=logs&j=456345f1-e486-5e62-b2a7-0064843ff30e)
- The attempt-1 task log was not finalized and is no longer retained.
- Azure reported the 180-minute timeout and unresponsive cancellation behavior.
- Failed-jobs attempt 2 reran `T: framework` on AcesShared 38 and succeeded from `2026-09-01T10:09:18Z` to `10:20:06Z`; build 15164848 completed successfully at `10:29:58Z`.
- Worker identity for the failed attempt is unavailable from retained authorized diagnostics.
### PR #26568
- Head: `4237a344488a1ab2c7cee633ec9b95fa8e9f7afb`
- Build: [15218580, attempt 1, `Build macOS tests` / `Build`](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15218580&view=logs&j=2ad83a99-f770-5ec5-96c6-63e16b4bb697)
- Worker: `AcesShared 4`
- Job started at `2026-09-04T20:10:11Z`; `Build` started at `20:17:12Z`.
- The `Build` task produced no finalized log. Azure failed the stage at `23:15:12Z` after the 180-minute cap and reported that the agent ignored cancellation for five minutes.
- `Package macOS tests` never ran, so the downstream macOS 27 test stage could not start.
- Retry is deferred until build 15218580 becomes terminal. Current usage is 0/25 PR lifetime and 0/10 for the current head.
First observed: `2026-09-01T07:00Z` (attempt-1 failure window; exact task completion unavailable because the agent stopped reporting).
Latest observed: `2026-09-04T23:15:12Z`.
Recurrence: **2 jobs across 2 distinct monitored PRs**.
## Classification and impact
This is a CI infrastructure/agent failure, not a source regression:
- It occurred in unrelated tasks (`Install dotnet workloads` and `Build`) on unrelated merge PRs.
- Both tasks stopped reporting without a finalized task log.
- Azure emitted the same job-limit and cancellation-timeout symptoms.
- The current PR changes only Xtro baseline files, and the same macOS build job previously completed successfully in about 36 minutes on build 15214043.
The failure blocks downstream stages and can suppress the diagnostic artifacts needed to identify a task-level cause.
## Investigation performed
- Inspected Azure timelines, task records, job issues, worker assignments, and available task logs.
- Compared the two stable signatures and rejected task-specific issues:
- #26212 tracks visibly slow Xcode downloads.
- #26226 tracks identified Provisionator `XcodeSelect` subprocess hangs.
- #26533 tracks an identified `xcodebuild -downloadPlatform tvOS` hang.
- Searched all open and closed `dotnet/macios` issues using the exact cancellation message, job names, 180-minute symptom, and missing-task-log evidence.
## Workaround
Rerun failed jobs after the build is terminal and normal head, mergeability, active-attempt, and retry-budget gates permit it. A durable fix needs agent-side diagnostics or watchdog handling that can survive an unresponsive task process and preserve the last task state.
Azure logs and artifacts are retention-limited.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Azure timelines, task records, job issues, worker assignments, and retained logs for builds 15164848 and 15218580; compare the two unresponsive-agent occurrences and the cited task-specific issues. Done means defining and implementing an agent-side diagnostic or watchdog approach that survives an unresponsive task, preserves the last task state, and allows cancellation and post-job diagnostics to complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100