dotnet / dotnet/macios

[CI] Sonoma corrupts mac-test-package.7z during test expansion

Open
#26,536 2 comments 0 reactions 0 assignees View on GitHub
ci-failure ci-postmortem
Dominant language
C#
Stars
2.9k
Forks
576
Avg merge
2d 13h
Merged PRs (30d)
96

Description

## Failure signature

The **X64 - Mac Sonoma (14)** `Expand tests` task can read `mac-test-package.7z`, but extraction reports widespread `ERROR: Data Error` entries and exits with code 2 on `XAMBOT-1049.Sonoma`. Other macOS lanes expand the same build artifact successfully.

Stable signature:

```text
X64 Mac Sonoma Expand tests extracts mac-test-package.7z with widespread ERROR: Data Error entries and exits 2 on XAMBOT-1049 while sibling workers expand the same artifact
```

## Confirmed occurrences

### PR #26481

- Head: `af51a02611066b9bd44db7a6ed8f1f1fc8069e70`
- [Build 15181206, Sonoma / Expand tests](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15181206&view=logs&j=623bc498-120b-5ff2-984a-5b073efffe1b&t=2d06de22-e976-5978-80f6-05dff7b36827)
- Worker: `XAMBOT-1049.Sonoma`
- Extraction produced **1,627** data errors and exited 2.
- Tahoe and Sequoia expanded the same artifact successfully. A targeted Sonoma retry on `XAMBOT-1006.Sonoma` succeeded, and the PR later merged.

### PR #26535

- Head: `3d2e0c0665e69b90d74f2b5d9a186d9b475a283f`
- [Build 15183044, Sonoma / Expand tests](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15183044&view=logs&j=623bc498-120b-5ff2-984a-5b073efffe1b&t=2d06de22-e976-5978-80f6-05dff7b36827)
- Worker: `XAMBOT-1049.Sonoma`
- Extraction produced **1,352** data errors and exited 2.
- `VSM-XAM-47.Tahoe.arm64` and `VSM-XAM-75.Sequoia.arm64` expanded the same artifact successfully in the same build.

First observed: `2026-09-02T07:06:24Z`.
Latest confirmed occurrence: `2026-09-02T12:30:22Z`.
Recurrence: **2 jobs across 2 distinct monitored PRs**.

## Classification and impact

This is a worker-local CI artifact transfer/storage corruption, not a source or deterministic test failure. The affected PRs are unrelated, the archive remains readable enough for 7-Zip to enumerate entries, and sibling workers successfully expand the same pipeline artifact.

The corruption prevents Sonoma tests from starting and fails the entire stage.

## Investigation performed

- Inspected Azure timelines, task logs, worker assignments, and 7-Zip summaries.
- Counted the retained `ERROR: Data Error` records for each occurrence.
- Compared extraction results across Sonoma, Tahoe, and Sequoia within the same builds.
- Verified the successful #26481 Sonoma retry on a different worker.
- Searched all open and closed `dotnet/macios` issues using the archive name, exact data-error text, task name, worker, and corruption terms. No matching issue existed.

## Workaround

After the full run is terminal and retry gates permit it, rerun the failed Sonoma stage so the artifact is transferred and expanded on a healthy worker.

Azure artifacts and task logs are retention-limited.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Azure timelines and task logs for builds 15181206 and 15183044, then compare the Sonoma worker assignments with the successful Tahoe, Sequoia, and Sonoma retry runs. Done means identifying the worker-local transfer or storage cause and establishing a durable remediation or documented mitigation for corrupted artifact expansion.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
ci-cd, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.