agent-substrate / agent-substrate/substrate

[P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated

Abierto
#644 1 comentario 0 reacciones 0 asignados Ver en GitHub
area/api kind/bug prio/P1
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

> Dedup note: verified against all open+closed issues and PRs 2026-07-30 —
> no prior report. Adjacent items ruled out: PR #558/#577 fixed the
> benchmark's *missing* credentials with mTLS pod certs (per-handshake
> reload) and never touch the ateclient bearer-token path; PR #248 /
> issue #222 introduced this token code without noting the no-refresh
> limitation; #459/PR #537 cover credbundle cert rotation (mTLS side only).

## Summary

`internal/ateclient` requests one ServiceAccount token with
`ExpirationSeconds: 3600` at dial time and wraps it in a **static**
per-RPC credential. The token is never re-minted. Any client process that
lives longer than one hour starts failing every RPC with
`Unauthenticated: invalid bearer token: jwt has expired` and never recovers
without a full reconnect.

## Evidence

- `internal/ateclient/builder.go:232-247` — `bearerTokenDialOption` mints the
token once; `bearerTokenCreds` is a `string` type whose
`GetRequestMetadata` (`builder.go:252`) returns the same literal token for
the lifetime of the connection.
- Empirically confirmed by the 2026-07-30 overnight soak
(`stability/overnight-test.yaml`, 8h run on GKE `substrate-poc`):
- Job started 05:42Z; first `jwt has expired` error at **06:49:18Z** (~1h07m in).
- From then until the run ended, **every** gRPC call failed:
176,707 `Unauthenticated` errors across ResumeActor / SuspendActor /
CycleResume, and all 8 `DeleteActor` cleanup calls failed, leaking the
soak actors.

## Impact

- The reference client library (used by `kubectl-ate`, `stability/stresstest`,
and as the model for external clients) cannot run a workload for more than
1 hour.
- Cleanup paths that run at process end (e.g. stresstest's actor deletion) are
guaranteed to fail in long runs, leaking actors and workers.
- Anyone copying this library pattern into a production client ships the
same time bomb.

## Reproduction

1. `ateclient.NewClient(...)` against any cluster.
2. Sleep 65 minutes.
3. Any RPC → `Unauthenticated: invalid bearer token: jwt has expired`.

## Suggested fix

Implement a refreshing `credentials.PerRPCCredentials`: cache the token and
its expiry, re-mint via the TokenRequest API when within a skew window
(e.g. 5 min) of expiry, under a mutex. `GetRequestMetadata` is already called
per-RPC, so refresh can be lazy and transparent.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.