dotnet / dotnet/macios

[CI] Windows Integration Dotnet tests exceeds the 120-minute job limit while progressing

Open
#26,543 7 comments 0 reactions 0 assignees View on GitHub
ci-postmortem copilot
Dominant language
C#
Stars
2.9k
Forks
576
Avg merge
2d 12h
Merged PRs (30d)
123

Description

## Summary

The **Windows Integration Tests / Dotnet tests** job has reached its fixed 120-minute Azure DevOps limit in two distinct monitored PRs while test work was either still progressing or had completed successfully. Azure then canceled the remaining remote-test or artifact-publication work.

- Stable signature: `Windows Integration Dotnet tests reaches the fixed 120-minute job limit while test workflow remains successful or actively progressing, canceling remaining test or publication work`
- Classification: CI infrastructure / job-duration budget failure
- Distinct monitored PRs: 2
- Confirmed occurrences: 2
- First confirmed occurrence: 2026-09-02T12:05:36Z
- Latest confirmed occurrence: 2026-09-02T17:46:07Z

## Confirmed occurrences

### PR #26020

- Head: `52aa6b725350ec79b8019d3d5e8523c817be9b27`
- Pipeline: `xamarin-macios-sim-pr-tests`
- [Build 15180927, Windows Integration attempt 2](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15180927&view=logs&j=9e1d9d57-f452-5f66-496f-9f10db70de3d&t=8807814c-90c9-5233-0d9d-54305cb46701)
- Worker: `VSM-XAM-136`
- Job interval: `2026-09-02T10:05:37Z`–`12:05:36Z`
- Download artifacts took 1,004.7 seconds, local tests took 619.0 seconds, and remote tests took 4,422.7 seconds. Those tasks succeeded, but the job limit canceled `Publish to Artifact Services Drop`.
- A full Windows Integration attempt 3 passed on `VSM-XAM-124`; build 15180927 succeeded and the PR later merged.

### PR #26513

- Head: `a354efde93df4b8e7ad9eb15a88ebd64916b2be1`
- Pipeline: `xamarin-macios-sim-pr-tests`
- [Build 15184608, attempt 1, Windows Integration / Dotnet tests](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15184608&view=logs&j=eac261bd-81a1-5958-df8d-94842fc987be&t=7eef1eb5-91f9-5641-dbf4-d80937931e01)
- Worker: `VSM-XAM-129`
- Job interval: `2026-09-02T15:45:52Z`–`17:46:07Z`
- `Run .NET tests remotely` ran from `16:37:57Z` until cancellation. Its log contains 31 completed passing test records and no test failures. The last recorded test passed at `17:44:20Z`, only 86 seconds before Azure began cancellation at `17:45:46Z`.
- Build 15184608 is still active, so no retry has been submitted. Repository policy requires rerunning the entire Windows Integration stage when eligible.

## Impact and reproduction pattern

The test workload duration varies enough that the 120-minute job limit can terminate otherwise healthy CI. Depending on timing, either tests remain incomplete or post-test result and artifact publication is skipped. This consumes the Windows and paired Mac agents for two hours and requires a full-stage retry.

## Investigation

The Azure build metadata, stage and job timelines, task durations, and remote-test logs were compared directly. Neither occurrence contains a deterministic test failure responsible for cancellation. The #26513 remote-test log continued reporting passes immediately before the deadline; #26020 had already completed both local and remote tests.

Separate searches across all open and closed issues used the exact Azure timeout message, stage and job names, `Run .NET tests remotely`, worker names, publication task, and Windows timeout terms. The only candidate was #26457, which is an unrelated Ventura `Show Bot Info` / `ioreg` pre-test stall.

## Current diagnosis and workaround

This is a CI duration-budget failure rather than a source regression. The current workaround is to rerun the entire Windows Integration stage after the parent build is terminal and retry policy permits it. Longer-term options are to increase the job budget, reduce setup and artifact-download time, or split the remote test workload so normal duration variance cannot consume the full job allowance.

Azure logs and artifacts are retention-limited.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Windows Integration / Dotnet tests job, its Azure build timeline, and the logs for Run .NET tests remotely and Publish to Artifact Services Drop. Compare the confirmed occurrences and determine a mitigation for the 120-minute limit; done means the stage completes its tests and artifact publication without deadline cancellation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.