dotnetup: transient CRL/revocation lookup failure hard-fails SDK bootstrap in CI (TimestampRevocationUnavailable)
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 1.3k
- PR merge metrics
- PR metrics pending
Description
## Summary
The `TestBuild: linux (arm64)` leg of a PR build failed during SDK bootstrap because **dotnetup's signature verification of the release metadata hard-failed on a transient certificate-revocation (CRL) lookup**. The CRL endpoint was unreachable from the build agent, and because dotnetup defaults to fail-closed revocation checking, the entire SDK install aborted — failing the build even though the PR change is unrelated (telemetry code).
This is an infrastructure fragility, not a code defect in the PR, but it flakes CI and will keep recurring on agents with transient CRL egress (arm64 legs seem especially prone).
## Where it happened
- PR: https://github.com/dotnet/sdk/pull/55127
- Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1492510
- Failing leg: `dotnet-sdk-public-ci (Build TestBuild: linux (arm64))`
- Failing step: `🟣 Build` — log https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/1492510/logs/68
- Only this single leg failed; windows / macOS / linux-x64 installed the same SDK fine.
## Error (from the build log)
```
Installing .NET SDK 11.0.100-preview.5.26227.104 to /mnt/vss/_work/1/s/.dotnet...
The following installs failed:
.NET SDK 11.0.100-preview.5.26227.104: Signature verification failed for
https://builds.dotnet.microsoft.com/dotnet/release-metadata/releases-index.json: 1 issue(s)
- TimestampRevocationUnavailable: timestamp authority chain could not confirm revocation
status (offline or unreachable). Detail: RevocationStatusUnknown:unable to get certificate CRL
| OfflineRevocation:unable to get certificate CRL
Error: Signature verification failed for .../releases-index.json: 1 issue(s)
- TimestampRevocationUnavailable: ...
Bash exited with code '1'.
```
Everything downstream is fallout from this (`🟣 Build` → bash exit 1 → `Publish Built Assets`: "Path does not exist: artifacts/packages/Release/Shipping" → `Monitor Helix Jobs`: "One or more non-monitor pipeline jobs failed").
## Root cause
dotnetup verifies the detached CMS signature of `releases-index.json` before installing, including a certificate-revocation check on the **timestamp authority** chain. Revocation checking defaults to fail-closed:
- `SignatureVerificationOptions.RevocationMode` defaults to `RevocationCheckMode.Online` — *"fail closed when CRL/OCSP is unreachable."*
`src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerificationOptions.cs:27-29`
- The chain maps `Online` → `X509RevocationMode.Online` with `RevocationFlag = EntireChain`.
`src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerifier.cs:787-794`
- When the chain reports `OfflineRevocation` / `RevocationStatusUnknown` (CRL unreachable), verification fails with `FailureCode.TimestampRevocationUnavailable`.
`src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerifier.cs:727`, `:847-852`
So a transient inability to fetch the CRL — not an actually-revoked cert — is treated as a fatal verification failure and aborts the install.
## Impact
- Flaky CI: any transient CRL/OCSP outage from a build/Helix agent fails the SDK bootstrap and the whole leg. Observed on `linux (arm64)`; likely to recur on other legs and on the automated `main → release/dnup` merge legs that also bootstrap via dotnetup.
- Real revocation is not distinguished from "couldn't reach the CRL," so genuinely transient network conditions look like security failures.
## Suggested mitigations (for discussion)
1. **Soft-fail revocation on unreachable CRL/OCSP.** Treat `OfflineRevocation` / `RevocationStatusUnknown` as a warning rather than a hard failure while still failing closed on an actual `Revoked` status. (i.e., don't let "couldn't reach CRL" be fatal, but keep "cert is revoked" fatal.)
2. **Bounded retry** around the revocation/verification step before failing (there is already a 30s `RevocationRetrievalTimeoutSeconds`, but a single attempt; a couple of retries would absorb brief outages).
3. **CI opt-in to `RevocationCheckMode.Offline`** for the SDK-bootstrap path, so CI uses locally cached CRLs and doesn't depend on live CRL egress from every agent, while end-user installs keep `Online`.
Option 1 (soft-fail on *unavailable*, hard-fail on *revoked*) is the most robust and matches how signing stacks like NuGet treat unreachable revocation.
## Workaround for now
Re-run the failed leg (`/azp run dotnet-sdk-public-ci` or AzDO "Rerun failed jobs" on build 1492510); the transient CRL outage will almost certainly not reproduce.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerificationOptions.cs:27-29 and SignatureVerifier.cs:727, 787-794, and 847-852; trace how unavailable CRL status becomes TimestampRevocationUnavailable. Compare the proposed soft-fail, retry, and offline-revocation options, then verify that transient unavailability no longer aborts bootstrap while an actual Revoked status remains fatal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100