dotnet / dotnet/macios

[CI] Azure DevOps TF10216 outage fails SAS, test-library, and publish tasks

Open
#26,224 4 comments 0 reactions 0 assignees View on GitHub
ci-failure ci-postmortem tests
Dominant language
C#
Stars
2.9k
Forks
576
Avg merge
2d 12h
Merged PRs (30d)
123

Description

## Failure signature

Azure DevOps-backed tasks fail before tests run with:

```text
TF10216: Azure DevOps services are currently unavailable. Try again later.
```

The same service outage affected unrelated tasks and PRs within minutes.

## Confirmed occurrences

| PR / head | Build and job | Failing task | Outcome |
|---|---|---|---|
| #26180 / `897e85a4b8e8e206c57ea59a7b0c3088abf18e15` | [14718322 - monotouch_ios, attempt 3](https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=14718322&view=logs&jobId=f184b60d-41ed-595a-0d93-d653d9488b65) | Generate BosStorageMirror SAS tokens | All task retries returned TF10216; the job failed. The stage attempt remains active. |
| #26207 / `84ada67506d107a5b8979d2bccb4aa1201b46d3b` | [14719170 - introspection, attempt 2](https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=14719170&view=logs&jobId=56850695-09ae-5157-9194-575d0b5f0ed0) | Download test libraries | TF10216 prevented the download; the job failed. The stage attempt remains active. |
| #26213 / `6acb80dc4985ae3a81ea8b6487ba2a3ec13148a5` | [14719548 - Publish results, attempt 2](https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=14719548&view=logs&jobId=b2f23583-7571-5539-20e5-c81c2127326b) | Download secrets / OIDC token | Repeated TF10216 responses failed the publish-results job. The build also had a separate checkout failure after the PR merged. |

First observed: 2026-07-21 23:59 UTC. Latest observed: 2026-07-22 00:04 UTC. Recurrence: 3 jobs across 3 distinct monitored PRs.

## Classification and impact

This is an Azure DevOps service/infrastructure outage, not a source or test regression. It blocks credential acquisition, artifact/test-library access, and result publication before the affected work can run.

## Investigation

- Inspected each Azure build timeline and failing task.
- Correlated the exact TF10216 response across separate agents, tasks, builds, and PRs.
- Searched open and closed `dotnet/macios` issues using TF10216, the full service-unavailable message, and the affected task names; no matching issue was found.

## Workaround

Retry failed jobs after Azure DevOps service health recovers, subject to mergeability and retry-budget gates. Do not submit another retry while an equivalent stage attempt is queued or running. Azure logs have limited retention.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the three linked Azure DevOps build timelines and their failing SAS, test-library, and publish tasks. Confirm the repeated TF10216 response across the listed jobs and check Azure DevOps service health before retrying after recovery; done means the affected jobs complete without the outage failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.