agent-substrate / agent-substrate/substrate

[P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated

Aperta
#644 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
area/api kind/bug prio/P1
Lingua principale
Go
Stelle
1.8k
Fork
316
Merge medio
2g 43m
PR unite (30g)
287

Descrizione

> Dedup note: verified against all open+closed issues and PRs 2026-07-30 —
> no prior report. Adjacent items ruled out: PR #558/#577 fixed the
> benchmark's *missing* credentials with mTLS pod certs (per-handshake
> reload) and never touch the ateclient bearer-token path; PR #248 /
> issue #222 introduced this token code without noting the no-refresh
> limitation; #459/PR #537 cover credbundle cert rotation (mTLS side only).

## Summary

`internal/ateclient` requests one ServiceAccount token with
`ExpirationSeconds: 3600` at dial time and wraps it in a **static**
per-RPC credential. The token is never re-minted. Any client process that
lives longer than one hour starts failing every RPC with
`Unauthenticated: invalid bearer token: jwt has expired` and never recovers
without a full reconnect.

## Evidence

- `internal/ateclient/builder.go:232-247` — `bearerTokenDialOption` mints the
token once; `bearerTokenCreds` is a `string` type whose
`GetRequestMetadata` (`builder.go:252`) returns the same literal token for
the lifetime of the connection.
- Empirically confirmed by the 2026-07-30 overnight soak
(`stability/overnight-test.yaml`, 8h run on GKE `substrate-poc`):
- Job started 05:42Z; first `jwt has expired` error at **06:49:18Z** (~1h07m in).
- From then until the run ended, **every** gRPC call failed:
176,707 `Unauthenticated` errors across ResumeActor / SuspendActor /
CycleResume, and all 8 `DeleteActor` cleanup calls failed, leaking the
soak actors.

## Impact

- The reference client library (used by `kubectl-ate`, `stability/stresstest`,
and as the model for external clients) cannot run a workload for more than
1 hour.
- Cleanup paths that run at process end (e.g. stresstest's actor deletion) are
guaranteed to fail in long runs, leaking actors and workers.
- Anyone copying this library pattern into a production client ships the
same time bomb.

## Reproduction

1. `ateclient.NewClient(...)` against any cluster.
2. Sleep 65 minutes.
3. Any RPC → `Unauthenticated: invalid bearer token: jwt has expired`.

## Suggested fix

Implement a refreshing `credentials.PerRPCCredentials`: cache the token and
its expiry, re-mint via the TokenRequest API when within a skew window
(e.g. 5 min) of expiry, under a mutex. `GetRequestMetadata` is already called
per-RPC, so refresh can be lazy and transparent.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.