agent-substrate / agent-substrate/substrate

[P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated

Offen
#644 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
area/api kind/bug prio/P1
Vorherrschende Sprache
Go
Sterne
1.8k
Forks
316
Ø Merge
2 T. 43 Min.
Gemergte PRs (30 T.)
287

Beschreibung

> Dedup note: verified against all open+closed issues and PRs 2026-07-30 —
> no prior report. Adjacent items ruled out: PR #558/#577 fixed the
> benchmark's *missing* credentials with mTLS pod certs (per-handshake
> reload) and never touch the ateclient bearer-token path; PR #248 /
> issue #222 introduced this token code without noting the no-refresh
> limitation; #459/PR #537 cover credbundle cert rotation (mTLS side only).

## Summary

`internal/ateclient` requests one ServiceAccount token with
`ExpirationSeconds: 3600` at dial time and wraps it in a **static**
per-RPC credential. The token is never re-minted. Any client process that
lives longer than one hour starts failing every RPC with
`Unauthenticated: invalid bearer token: jwt has expired` and never recovers
without a full reconnect.

## Evidence

- `internal/ateclient/builder.go:232-247` — `bearerTokenDialOption` mints the
token once; `bearerTokenCreds` is a `string` type whose
`GetRequestMetadata` (`builder.go:252`) returns the same literal token for
the lifetime of the connection.
- Empirically confirmed by the 2026-07-30 overnight soak
(`stability/overnight-test.yaml`, 8h run on GKE `substrate-poc`):
- Job started 05:42Z; first `jwt has expired` error at **06:49:18Z** (~1h07m in).
- From then until the run ended, **every** gRPC call failed:
176,707 `Unauthenticated` errors across ResumeActor / SuspendActor /
CycleResume, and all 8 `DeleteActor` cleanup calls failed, leaking the
soak actors.

## Impact

- The reference client library (used by `kubectl-ate`, `stability/stresstest`,
and as the model for external clients) cannot run a workload for more than
1 hour.
- Cleanup paths that run at process end (e.g. stresstest's actor deletion) are
guaranteed to fail in long runs, leaking actors and workers.
- Anyone copying this library pattern into a production client ships the
same time bomb.

## Reproduction

1. `ateclient.NewClient(...)` against any cluster.
2. Sleep 65 minutes.
3. Any RPC → `Unauthenticated: invalid bearer token: jwt has expired`.

## Suggested fix

Implement a refreshing `credentials.PerRPCCredentials`: cache the token and
its expiry, re-mint via the TokenRequest API when within a skew window
(e.g. 5 min) of expiry, under a mutex. `GetRequestMetadata` is already called
per-RPC, so refresh can be lazy and transparent.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.