DCR credential resolution never refreshes after a replica's startup, allowing cross-replica generation drift

Summary

Each replica resolves its outbound DCR (Dynamic Client Registration) credentials for an upstream OAuth provider exactly once, at startup (buildUpstreamConfigs, called from the EmbeddedAuthServer constructor in pkg/authserver/runner/embeddedauthserver.go). The resolved client_id/client_secret are baked into the replica's in-memory OAuth2Config for the life of the process. Nothing ever re-resolves or refreshes it.

Separately, the Redis-backed DCRCredentialStore ties a cached credential's TTL to the upstream-supplied client_secret_expires_at (pkg/authserver/storage/redis.go, since #5195).

Combined, this means: if the upstream issues a time-limited client secret, and that TTL lapses while a replica is still running, a later-starting replica (rolling deploy straggler, autoscale, crash-restart) will see a cache miss, register a new client with the upstream ("generation B"), while the still-running replica keeps using its original registration ("generation A") for the rest of its life. Both generations are then live simultaneously behind the load balancer.

Failure mode

A user's login flow can start on a generation-A replica (redirected to the upstream using generation A's client_id) and have its callback land on a generation-B replica (no session affinity across the OAuth redirect). The generation-B replica tries to redeem the authorization code with its own (different) client credentials, which the upstream rejects — typically invalid_grant/unauthorized_client. This is a visible login failure for the affected request, not a silent security issue.

Reachability

Only occurs when:

1. The upstream IdP sets a non-zero client_secret_expires_at on DCR registration (common for enterprise IdPs, not universal — many dev/test and some production IdPs issue non-expiring secrets, in which case this can't happen at all), AND
2. Replicas are restarted/added at staggered times relative to each other, straddling that expiry.

Why this isn't a quick fix

This needs one of:

• A scheduled re-resolution/refresh task tied to the credential's expiry, with safe hot-swapping of the in-memory OAuth2Config under concurrent use, or
• A push-based invalidation mechanism (e.g. pub/sub) so a replica that (re)registers tells its siblings to refresh, or
• Avoiding the problem by proactively rotating/refreshing well before expiry from one coordinated place, rather than reactively on whichever replica happens to hit a cache miss first.

All three require new infrastructure beyond the existing single-shot-resolve-at-startup architecture — not a bounded fix to the storage layer.

Origin

Flagged during review of #6474 (a fix for a different, narrower DCR cache-fill race — the simultaneous-first-fill race between replicas starting at the same time, fixed via PutIfAbsent/create-if-absent semantics). Confirmed via git log that both the single-shot-resolution architecture (#3540, #5044) and the TTL/expiry mechanism (#5195) predate #6474 by a wide margin — #6474 did not introduce or worsen this gap, it only fixed a different race in the same area.

Suggested first step

Before designing the full fix, check whether the upstream's invalid_grant/unauthorized_client rejection currently surfaces as a clear, actionable error to the end user/operator, or gets swallowed into a generic failure — a "fail loudly with a clear re-authenticate message" may be a reasonable interim mitigation even before real cross-replica refresh coordination is built.

Start with buildUpstreamConfigs and the EmbeddedAuthServer constructor in pkg/authserver/runner/embeddedauthserver.go, then trace DCRCredentialStore and its TTL handling in pkg/authserver/storage/redis.go. Inspect how upstream invalid_grant or unauthorized_client failures reach users and operators. Done would require an agreed approach for cross-replica credential refresh, or a clearly actionable interim failure message.