agent-substrate / agent-substrate/substrate

[P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated

Đang mở
#644 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/api kind/bug prio/P1
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

> Dedup note: verified against all open+closed issues and PRs 2026-07-30 —
> no prior report. Adjacent items ruled out: PR #558/#577 fixed the
> benchmark's *missing* credentials with mTLS pod certs (per-handshake
> reload) and never touch the ateclient bearer-token path; PR #248 /
> issue #222 introduced this token code without noting the no-refresh
> limitation; #459/PR #537 cover credbundle cert rotation (mTLS side only).

## Summary

`internal/ateclient` requests one ServiceAccount token with
`ExpirationSeconds: 3600` at dial time and wraps it in a **static**
per-RPC credential. The token is never re-minted. Any client process that
lives longer than one hour starts failing every RPC with
`Unauthenticated: invalid bearer token: jwt has expired` and never recovers
without a full reconnect.

## Evidence

- `internal/ateclient/builder.go:232-247` — `bearerTokenDialOption` mints the
token once; `bearerTokenCreds` is a `string` type whose
`GetRequestMetadata` (`builder.go:252`) returns the same literal token for
the lifetime of the connection.
- Empirically confirmed by the 2026-07-30 overnight soak
(`stability/overnight-test.yaml`, 8h run on GKE `substrate-poc`):
- Job started 05:42Z; first `jwt has expired` error at **06:49:18Z** (~1h07m in).
- From then until the run ended, **every** gRPC call failed:
176,707 `Unauthenticated` errors across ResumeActor / SuspendActor /
CycleResume, and all 8 `DeleteActor` cleanup calls failed, leaking the
soak actors.

## Impact

- The reference client library (used by `kubectl-ate`, `stability/stresstest`,
and as the model for external clients) cannot run a workload for more than
1 hour.
- Cleanup paths that run at process end (e.g. stresstest's actor deletion) are
guaranteed to fail in long runs, leaking actors and workers.
- Anyone copying this library pattern into a production client ships the
same time bomb.

## Reproduction

1. `ateclient.NewClient(...)` against any cluster.
2. Sleep 65 minutes.
3. Any RPC → `Unauthenticated: invalid bearer token: jwt has expired`.

## Suggested fix

Implement a refreshing `credentials.PerRPCCredentials`: cache the token and
its expiry, re-mint via the TokenRequest API when within a skew window
(e.g. 5 min) of expiry, under a mutex. `GetRequestMetadata` is already called
per-RPC, so refresh can be lazy and transparent.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.