Internal pipeline: transient macOS notarization flake fails native build+sign on main
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 201
Description
## Summary
The internal `microsoft-aspire` pipeline (definition 1602) intermittently fails on
main in the **Build+Sign native packages** stage. A transient macOS notarization
(ESRP/Apple) flake fails the *entire* sign task even though a plain retry recovers
and notarization actually succeeds.
## Where it happens
- Pipeline: `microsoft-aspire` (dnceng/internal, definition 1602), macOS jobs of the
native build+sign stage.
- Both macOS steps that notarize are affected: `🟣Sign aspire-managed` (notarizes the
`aspire-managed` binary) and `🟣Build native packages` (notarizes the native `aspire`
CLI binary).
- Example failure: build `20260708.1` (buildId **3017414**), `osx_x64` job,
`🟣Sign aspire-managed`.
## Expected vs. actual
- **Expected:** a transient notarization attempt failure is absorbed by the SignTool's
built-in retry (up to 5 attempts), and the build passes.
- **Actual:** the task fails despite the retry succeeding. In build 3017414, notarization
attempt 1 flaked, attempt 2 **succeeded with 0 errors**, yet the job failed with:
```
/Users/runner/.nuget/packages/microsoft.dotnet.arcade.sdk/10.0.0-beta.26324.4/tools/Sign.proj(76,5): error : Failed to execute MSBuild on the project file '/Users/runner/work/1/s/artifacts/tmp/Release/Signing/Round0-Notarize.proj' with exit code '1'.
```
The downstream `Publish native symbols` / `Publish Artifacts` failures in the same job
are only a consequence of the signed binary never being produced.
## What the notarization actually returned
Digging into the signing binlog (`NotarizationRound0-Attempt1.binlog`) for the failed
attempt, ESRP returned:
```
OperationId: fa30fdf8-... failed. Status: FailDoNotRetry
Error: {"code":"MacSignFailed","details":{"errorDetails":""},"innerError":null}
FilePath: .../osx-x64/publish/aspire-managed (149.32 MB)
```
Notable:
- The error is an opaque **`MacSignFailed`** with **empty `errorDetails`** — no Apple-side
reason is surfaced.
- ESRP marked it **`FailDoNotRetry`**, yet the very next attempt resubmitted a fresh
operation and **succeeded** — so the failure is transient and the "do not retry"
classification is misleading.
## Pattern across recent builds
- `3017414`: `osx-x64`, `🟣Sign aspire-managed` — notary flake → build failed.
- `3015220`: `osx-arm64`, `🟣Build native packages` — same `Sign.proj(76,5)` failure → build failed.
- `3016782`, `3016108`, `3016049`, `3015923`: notarization clean, builds ok.
So the failure is **not arch-specific** (hits x64 and arm64) and **not step-specific**
(hits both notarizing steps), it's an intermittent ESRP/Apple notarization flake on the
large (~149 MB) self-contained binaries.
## Root cause & upstream fix
The `Microsoft.DotNet.SignTool` package aspire consumes (pinned to the Arcade SDK version,
built from arcade `release/10.0`) calls MSBuild's `LogError` on an intermediate
notarization attempt. MSBuild marks a task failed once `LogError` is called, even if the
task later returns success — so any single flaky attempt fails the whole build.
The fix (`dotnet/arcade#16669`, which suppresses errors on non-final attempts) landed on
arcade `main` but was never backported to `release/10.0`. The buggy retry loop itself
*was* backported (`dotnet/arcade#16586`).
Backport of the fix to `release/10.0`: **dotnet/arcade#17090**. This issue will resolve
once that merges and a new arcade build flows in.
Contributor guide
Research direction
Start by tracking dotnet/arcade#17090, the backport of the SignTool retry fix to release/10.0. Confirm that it merges and that a new Arcade SDK build reaches Aspire; then verify the macOS Build+Sign native packages stage passes for both notarizing steps. There is no Aspire file named for a direct change in this issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, macos
- Domain
- build-system, ci-cd
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 20/100