[CI] Sonoma corrupts mac-test-package.7z during test expansion
- Dominant language
- C#
- Stars
- 2.9k
- Forks
- 576
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 96
Description
## Failure signature
The **X64 - Mac Sonoma (14)** `Expand tests` task can read `mac-test-package.7z`, but extraction reports widespread `ERROR: Data Error` entries and exits with code 2 on `XAMBOT-1049.Sonoma`. Other macOS lanes expand the same build artifact successfully.
Stable signature:
```text
X64 Mac Sonoma Expand tests extracts mac-test-package.7z with widespread ERROR: Data Error entries and exits 2 on XAMBOT-1049 while sibling workers expand the same artifact
```
## Confirmed occurrences
### PR #26481
- Head: `af51a02611066b9bd44db7a6ed8f1f1fc8069e70`
- [Build 15181206, Sonoma / Expand tests](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15181206&view=logs&j=623bc498-120b-5ff2-984a-5b073efffe1b&t=2d06de22-e976-5978-80f6-05dff7b36827)
- Worker: `XAMBOT-1049.Sonoma`
- Extraction produced **1,627** data errors and exited 2.
- Tahoe and Sequoia expanded the same artifact successfully. A targeted Sonoma retry on `XAMBOT-1006.Sonoma` succeeded, and the PR later merged.
### PR #26535
- Head: `3d2e0c0665e69b90d74f2b5d9a186d9b475a283f`
- [Build 15183044, Sonoma / Expand tests](https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=15183044&view=logs&j=623bc498-120b-5ff2-984a-5b073efffe1b&t=2d06de22-e976-5978-80f6-05dff7b36827)
- Worker: `XAMBOT-1049.Sonoma`
- Extraction produced **1,352** data errors and exited 2.
- `VSM-XAM-47.Tahoe.arm64` and `VSM-XAM-75.Sequoia.arm64` expanded the same artifact successfully in the same build.
First observed: `2026-09-02T07:06:24Z`.
Latest confirmed occurrence: `2026-09-02T12:30:22Z`.
Recurrence: **2 jobs across 2 distinct monitored PRs**.
## Classification and impact
This is a worker-local CI artifact transfer/storage corruption, not a source or deterministic test failure. The affected PRs are unrelated, the archive remains readable enough for 7-Zip to enumerate entries, and sibling workers successfully expand the same pipeline artifact.
The corruption prevents Sonoma tests from starting and fails the entire stage.
## Investigation performed
- Inspected Azure timelines, task logs, worker assignments, and 7-Zip summaries.
- Counted the retained `ERROR: Data Error` records for each occurrence.
- Compared extraction results across Sonoma, Tahoe, and Sequoia within the same builds.
- Verified the successful #26481 Sonoma retry on a different worker.
- Searched all open and closed `dotnet/macios` issues using the archive name, exact data-error text, task name, worker, and corruption terms. No matching issue existed.
## Workaround
After the full run is terminal and retry gates permit it, rerun the failed Sonoma stage so the artifact is transferred and expanded on a healthy worker.
Azure artifacts and task logs are retention-limited.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Azure timelines and task logs for builds 15181206 and 15183044, then compare the Sonoma worker assignments with the successful Tahoe, Sequoia, and Sonoma retry runs. Done means identifying the worker-local transfer or storage cause and establishing a durable remediation or documented mitigation for corrupted artifact expansion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- ci-cd, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100