dotnet / dotnet/sdk

dotnetup: transient CRL/revocation lookup failure hard-fails SDK bootstrap in CI (TimestampRevocationUnavailable)

Open
#55,135 2 comments 0 reactions 0 assignees View on GitHub
Area-dotnetup untriaged
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

## Summary

The `TestBuild: linux (arm64)` leg of a PR build failed during SDK bootstrap because **dotnetup's signature verification of the release metadata hard-failed on a transient certificate-revocation (CRL) lookup**. The CRL endpoint was unreachable from the build agent, and because dotnetup defaults to fail-closed revocation checking, the entire SDK install aborted — failing the build even though the PR change is unrelated (telemetry code).

This is an infrastructure fragility, not a code defect in the PR, but it flakes CI and will keep recurring on agents with transient CRL egress (arm64 legs seem especially prone).

## Where it happened

- PR: https://github.com/dotnet/sdk/pull/55127
- Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1492510
- Failing leg: `dotnet-sdk-public-ci (Build TestBuild: linux (arm64))`
- Failing step: `🟣 Build` — log https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/1492510/logs/68
- Only this single leg failed; windows / macOS / linux-x64 installed the same SDK fine.

## Error (from the build log)

```
Installing .NET SDK 11.0.100-preview.5.26227.104 to /mnt/vss/_work/1/s/.dotnet...
The following installs failed:
.NET SDK 11.0.100-preview.5.26227.104: Signature verification failed for
https://builds.dotnet.microsoft.com/dotnet/release-metadata/releases-index.json: 1 issue(s)
- TimestampRevocationUnavailable: timestamp authority chain could not confirm revocation
status (offline or unreachable). Detail: RevocationStatusUnknown:unable to get certificate CRL
| OfflineRevocation:unable to get certificate CRL
Error: Signature verification failed for .../releases-index.json: 1 issue(s)
- TimestampRevocationUnavailable: ...
Bash exited with code '1'.
```

Everything downstream is fallout from this (`🟣 Build` → bash exit 1 → `Publish Built Assets`: "Path does not exist: artifacts/packages/Release/Shipping" → `Monitor Helix Jobs`: "One or more non-monitor pipeline jobs failed").

## Root cause

dotnetup verifies the detached CMS signature of `releases-index.json` before installing, including a certificate-revocation check on the **timestamp authority** chain. Revocation checking defaults to fail-closed:

- `SignatureVerificationOptions.RevocationMode` defaults to `RevocationCheckMode.Online` — *"fail closed when CRL/OCSP is unreachable."*
`src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerificationOptions.cs:27-29`
- The chain maps `Online` → `X509RevocationMode.Online` with `RevocationFlag = EntireChain`.
`src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerifier.cs:787-794`
- When the chain reports `OfflineRevocation` / `RevocationStatusUnknown` (CRL unreachable), verification fails with `FailureCode.TimestampRevocationUnavailable`.
`src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerifier.cs:727`, `:847-852`

So a transient inability to fetch the CRL — not an actually-revoked cert — is treated as a fatal verification failure and aborts the install.

## Impact

- Flaky CI: any transient CRL/OCSP outage from a build/Helix agent fails the SDK bootstrap and the whole leg. Observed on `linux (arm64)`; likely to recur on other legs and on the automated `main → release/dnup` merge legs that also bootstrap via dotnetup.
- Real revocation is not distinguished from "couldn't reach the CRL," so genuinely transient network conditions look like security failures.

## Suggested mitigations (for discussion)

1. **Soft-fail revocation on unreachable CRL/OCSP.** Treat `OfflineRevocation` / `RevocationStatusUnknown` as a warning rather than a hard failure while still failing closed on an actual `Revoked` status. (i.e., don't let "couldn't reach CRL" be fatal, but keep "cert is revoked" fatal.)
2. **Bounded retry** around the revocation/verification step before failing (there is already a 30s `RevocationRetrievalTimeoutSeconds`, but a single attempt; a couple of retries would absorb brief outages).
3. **CI opt-in to `RevocationCheckMode.Offline`** for the SDK-bootstrap path, so CI uses locally cached CRLs and doesn't depend on live CRL egress from every agent, while end-user installs keep `Online`.

Option 1 (soft-fail on *unavailable*, hard-fail on *revoked*) is the most robust and matches how signing stacks like NuGet treat unreachable revocation.

## Workaround for now

Re-run the failed leg (`/azp run dotnet-sdk-public-ci` or AzDO "Rerun failed jobs" on build 1492510); the transient CRL outage will almost certainly not reproduce.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/Installer/Microsoft.Dotnet.Installation/Internal/Signing/SignatureVerificationOptions.cs:27-29 and SignatureVerifier.cs:727, 787-794, and 847-852; trace how unavailable CRL status becomes TimestampRevocationUnavailable. Compare the proposed soft-fail, retry, and offline-revocation options, then verify that transient unavailability no longer aborts bootstrap while an actual Revoked status remains fatal.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.