agent-substrate / agent-substrate/substrate
[P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated
- Lenguaje dominante
- Go
- Estrellas
- 1.8k
- Forks
- 316
- Merge medio
- 2 d 43 min
- PR fusionados (30 d)
- 287
Descripción
> Dedup note: verified against all open+closed issues and PRs 2026-07-30 —
> no prior report. Adjacent items ruled out: PR #558/#577 fixed the
> benchmark's *missing* credentials with mTLS pod certs (per-handshake
> reload) and never touch the ateclient bearer-token path; PR #248 /
> issue #222 introduced this token code without noting the no-refresh
> limitation; #459/PR #537 cover credbundle cert rotation (mTLS side only).
## Summary
`internal/ateclient` requests one ServiceAccount token with
`ExpirationSeconds: 3600` at dial time and wraps it in a **static**
per-RPC credential. The token is never re-minted. Any client process that
lives longer than one hour starts failing every RPC with
`Unauthenticated: invalid bearer token: jwt has expired` and never recovers
without a full reconnect.
## Evidence
- `internal/ateclient/builder.go:232-247` — `bearerTokenDialOption` mints the
token once; `bearerTokenCreds` is a `string` type whose
`GetRequestMetadata` (`builder.go:252`) returns the same literal token for
the lifetime of the connection.
- Empirically confirmed by the 2026-07-30 overnight soak
(`stability/overnight-test.yaml`, 8h run on GKE `substrate-poc`):
- Job started 05:42Z; first `jwt has expired` error at **06:49:18Z** (~1h07m in).
- From then until the run ended, **every** gRPC call failed:
176,707 `Unauthenticated` errors across ResumeActor / SuspendActor /
CycleResume, and all 8 `DeleteActor` cleanup calls failed, leaking the
soak actors.
## Impact
- The reference client library (used by `kubectl-ate`, `stability/stresstest`,
and as the model for external clients) cannot run a workload for more than
1 hour.
- Cleanup paths that run at process end (e.g. stresstest's actor deletion) are
guaranteed to fail in long runs, leaking actors and workers.
- Anyone copying this library pattern into a production client ships the
same time bomb.
## Reproduction
1. `ateclient.NewClient(...)` against any cluster.
2. Sleep 65 minutes.
3. Any RPC → `Unauthenticated: invalid bearer token: jwt has expired`.
## Suggested fix
Implement a refreshing `credentials.PerRPCCredentials`: cache the token and
its expiry, re-mint via the TokenRequest API when within a skew window
(e.g. 5 min) of expiry, under a mutex. `GetRequestMetadata` is already called
per-RPC, so refresh can be lazy and transparent.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.