GCP ADC token refresh has no retry on transient 5xx, bypassing wrapTokenSource

Summary

When a GCP deploy authenticates via Application Default Credentials (ADC) — which is how CI authenticates (google-github-actions/auth sets GOOGLE_APPLICATION_CREDENTIALS) — a transient 5xx from the STS / token endpoint is not retried, and surfaces to in-flight RPCs as a fatal Unauthenticated, aborting an otherwise-healthy deploy.

Observed in the gcp BYOC smoketest (defang-mvp run 29755281369):

Error: failed to poll build operation: rpc error: code = Unauthenticated
desc = transport: per-RPC creds failed due to error: credentials: status code 503:
upstream connect error or disconnect/reset before headers ...
delayed connect error: Connection refused

(A rerun passed — it was a one-off blip.)

Root cause

wrapTokenSource / retryingTokenSource (in pkg/clouds/gcp/tokensource.go) adds transient-error retry around token refresh, and its stated purpose is precisely to "prevent a slow oauth2 response from surfacing to in-flight RPCs as a fatal Unauthenticated error during long deploys."

But it is only installed on the credential sources that login.go builds explicitly (interactive login, stored creds, and the GitHub-OIDC/WIF source) via option.WithTokenSource. Path 1 (ADC) returns early without wrapping:

// No need to pass down ADC token source via options since ADC is automatically used by gcp sdk
return nil

So under ADC the gcp SDK drives its own cloud.google.com/go/auth credentials and does its own STS token exchange, with no transient-5xx retry — our wrapper is not in the path. The credentials: status code 503 prefix (from cloud.google.com/go/auth's stsexchange, not x/oauth2) confirms the failing exchange happened inside the SDK's own ADC flow.

Current mitigation

• PR #2184 — retries the cloudbuild poll once when this transient auth-transport 503 shows up as Unauthenticated. This is what actually covers the CI/ADC path today, but it is a symptom-level guard scoped to build polling.
• PR #2185 — hardens isTransientTokenError to retry 5xx/429/408, covering the local interactive/stored-cred paths that flow through wrapTokenSource. Does not cover ADC.

Neither gives ADC deploys a general token-refresh retry: any other RPC (not just build polling) hit by a transient token 5xx under ADC would still fail hard.

Options for a real fix

1. Always route ADC through our wrapper — resolve ADC into an explicit oauth2.TokenSource (e.g. google.FindDefaultCredentials / credentials.DetectDefault), wrap it with wrapTokenSource, and pass it via option.WithTokenSource, instead of returning nil. Gives every RPC the same transient-5xx retry regardless of credential source. Main cost: we take over ADC resolution rather than letting the SDK do it implicitly.
2. Configure SDK-level retry — supply a gax retry/WithGRPCDialOption policy on gcp.Options that treats Unauthenticated-with-transport-markers as retryable. More invasive and easy to over-retry genuine auth failures.
3. Leave the poll-layer guard as-is and accept that non-build RPCs under ADC remain unprotected. Lowest effort; leaves the general gap open.

Leaning toward option 1: it's the smallest change that closes the gap uniformly and reuses the retry logic we already have.

Acceptance

• A transient 5xx during token refresh under ADC is retried rather than aborting the deploy.
• Behavior verified by a unit test that exercises the ADC credential path through the retry wrapper.

Trace the ADC early return in login.go and the existing wrapTokenSource/retryingTokenSource implementation in pkg/clouds/gcp/tokensource.go. Start by examining how the GCP credential source is constructed and how existing token-source tests model transient errors. Done means ADC credentials pass through the retry wrapper and a unit test verifies that a transient 5xx during refresh is retried.