[CI] Windows Integration Dotnet tests exceeds the 120-minute job limit while progressing
- Dominant language
- C#
- Stars
- 2.9k
- Forks
- 576
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 123
Description
## Summary
The **Windows Integration Tests / Dotnet tests** job has reached its fixed 120-minute Azure DevOps limit in two distinct monitored PRs while test work was either still progressing or had completed successfully. Azure then canceled the remaining remote-test or artifact-publication work.
- Stable signature: `Windows Integration Dotnet tests reaches the fixed 120-minute job limit while test workflow remains successful or actively progressing, canceling remaining test or publication work`
- Classification: CI infrastructure / job-duration budget failure
- Distinct monitored PRs: 2
- Confirmed occurrences: 2
- First confirmed occurrence: 2026-09-02T12:05:36Z
- Latest confirmed occurrence: 2026-09-02T17:46:07Z
## Confirmed occurrences
### PR #26020
- Head: `52aa6b725350ec79b8019d3d5e8523c817be9b27`
- Pipeline: `xamarin-macios-sim-pr-tests`
- [Build 15180927, Windows Integration attempt 2](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15180927&view=logs&j=9e1d9d57-f452-5f66-496f-9f10db70de3d&t=8807814c-90c9-5233-0d9d-54305cb46701)
- Worker: `VSM-XAM-136`
- Job interval: `2026-09-02T10:05:37Z`–`12:05:36Z`
- Download artifacts took 1,004.7 seconds, local tests took 619.0 seconds, and remote tests took 4,422.7 seconds. Those tasks succeeded, but the job limit canceled `Publish to Artifact Services Drop`.
- A full Windows Integration attempt 3 passed on `VSM-XAM-124`; build 15180927 succeeded and the PR later merged.
### PR #26513
- Head: `a354efde93df4b8e7ad9eb15a88ebd64916b2be1`
- Pipeline: `xamarin-macios-sim-pr-tests`
- [Build 15184608, attempt 1, Windows Integration / Dotnet tests](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15184608&view=logs&j=eac261bd-81a1-5958-df8d-94842fc987be&t=7eef1eb5-91f9-5641-dbf4-d80937931e01)
- Worker: `VSM-XAM-129`
- Job interval: `2026-09-02T15:45:52Z`–`17:46:07Z`
- `Run .NET tests remotely` ran from `16:37:57Z` until cancellation. Its log contains 31 completed passing test records and no test failures. The last recorded test passed at `17:44:20Z`, only 86 seconds before Azure began cancellation at `17:45:46Z`.
- Build 15184608 is still active, so no retry has been submitted. Repository policy requires rerunning the entire Windows Integration stage when eligible.
## Impact and reproduction pattern
The test workload duration varies enough that the 120-minute job limit can terminate otherwise healthy CI. Depending on timing, either tests remain incomplete or post-test result and artifact publication is skipped. This consumes the Windows and paired Mac agents for two hours and requires a full-stage retry.
## Investigation
The Azure build metadata, stage and job timelines, task durations, and remote-test logs were compared directly. Neither occurrence contains a deterministic test failure responsible for cancellation. The #26513 remote-test log continued reporting passes immediately before the deadline; #26020 had already completed both local and remote tests.
Separate searches across all open and closed issues used the exact Azure timeout message, stage and job names, `Run .NET tests remotely`, worker names, publication task, and Windows timeout terms. The only candidate was #26457, which is an unrelated Ventura `Show Bot Info` / `ioreg` pre-test stall.
## Current diagnosis and workaround
This is a CI duration-budget failure rather than a source regression. The current workaround is to rerun the entire Windows Integration stage after the parent build is terminal and retry policy permits it. Longer-term options are to increase the job budget, reduce setup and artifact-download time, or split the remote test workload so normal duration variance cannot consume the full job allowance.
Azure logs and artifacts are retention-limited.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Windows Integration / Dotnet tests job, its Azure build timeline, and the logs for Run .NET tests remotely and Publish to Artifact Services Drop. Compare the confirmed occurrences and determine a mitigation for the 120-minute limit; done means the stage completes its tests and artifact publication without deadline cancellation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100