microsoft / microsoft/aspire

Internal pipeline: transient macOS notarization flake fails native build+sign on main

Open
#18,715 0 comments 0 reactions 0 assignees View on GitHub
area-engineering-systems
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 12h
Merged PRs (30d)
201

Description

## Summary

The internal `microsoft-aspire` pipeline (definition 1602) intermittently fails on
main in the **Build+Sign native packages** stage. A transient macOS notarization
(ESRP/Apple) flake fails the *entire* sign task even though a plain retry recovers
and notarization actually succeeds.

## Where it happens

- Pipeline: `microsoft-aspire` (dnceng/internal, definition 1602), macOS jobs of the
native build+sign stage.
- Both macOS steps that notarize are affected: `🟣Sign aspire-managed` (notarizes the
`aspire-managed` binary) and `🟣Build native packages` (notarizes the native `aspire`
CLI binary).
- Example failure: build `20260708.1` (buildId **3017414**), `osx_x64` job,
`🟣Sign aspire-managed`.

## Expected vs. actual

- **Expected:** a transient notarization attempt failure is absorbed by the SignTool's
built-in retry (up to 5 attempts), and the build passes.
- **Actual:** the task fails despite the retry succeeding. In build 3017414, notarization
attempt 1 flaked, attempt 2 **succeeded with 0 errors**, yet the job failed with:

```
/Users/runner/.nuget/packages/microsoft.dotnet.arcade.sdk/10.0.0-beta.26324.4/tools/Sign.proj(76,5): error : Failed to execute MSBuild on the project file '/Users/runner/work/1/s/artifacts/tmp/Release/Signing/Round0-Notarize.proj' with exit code '1'.
```

The downstream `Publish native symbols` / `Publish Artifacts` failures in the same job
are only a consequence of the signed binary never being produced.

## What the notarization actually returned

Digging into the signing binlog (`NotarizationRound0-Attempt1.binlog`) for the failed
attempt, ESRP returned:

```
OperationId: fa30fdf8-... failed. Status: FailDoNotRetry
Error: {"code":"MacSignFailed","details":{"errorDetails":""},"innerError":null}
FilePath: .../osx-x64/publish/aspire-managed (149.32 MB)
```

Notable:
- The error is an opaque **`MacSignFailed`** with **empty `errorDetails`** — no Apple-side
reason is surfaced.
- ESRP marked it **`FailDoNotRetry`**, yet the very next attempt resubmitted a fresh
operation and **succeeded** — so the failure is transient and the "do not retry"
classification is misleading.

## Pattern across recent builds

- `3017414`: `osx-x64`, `🟣Sign aspire-managed` — notary flake → build failed.
- `3015220`: `osx-arm64`, `🟣Build native packages` — same `Sign.proj(76,5)` failure → build failed.
- `3016782`, `3016108`, `3016049`, `3015923`: notarization clean, builds ok.

So the failure is **not arch-specific** (hits x64 and arm64) and **not step-specific**
(hits both notarizing steps), it's an intermittent ESRP/Apple notarization flake on the
large (~149 MB) self-contained binaries.

## Root cause & upstream fix

The `Microsoft.DotNet.SignTool` package aspire consumes (pinned to the Arcade SDK version,
built from arcade `release/10.0`) calls MSBuild's `LogError` on an intermediate
notarization attempt. MSBuild marks a task failed once `LogError` is called, even if the
task later returns success — so any single flaky attempt fails the whole build.

The fix (`dotnet/arcade#16669`, which suppresses errors on non-final attempts) landed on
arcade `main` but was never backported to `release/10.0`. The buggy retry loop itself
*was* backported (`dotnet/arcade#16586`).

Backport of the fix to `release/10.0`: **dotnet/arcade#17090**. This issue will resolve
once that merges and a new arcade build flows in.

Contributor guide

Open the contributing guide

Research direction

Start by tracking dotnet/arcade#17090, the backport of the SignTool retry fix to release/10.0. Confirm that it merges and that a new Arcade SDK build reaches Aspire; then verify the macOS Build+Sign native packages stage passes for both notarizing steps. There is no Aspire file named for a direct change in this issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, macos
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.