cubestore: web identity S3 credentials are only refreshed when the token file changes, so they expire after ~1h
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
### Summary
`spawn_creds_refresh_loop` in `remotefs/s3.rs` does refresh S3 credentials, but in web identity mode it re-derives them **only when the token file's mtime changes**:
```rust
// In web identity mode, only refresh when the token file changed.
if let (Some(ref file), Some(_)) = (&token_file, &role_arn) {
let current_modified = std::fs::metadata(file).ok().and_then(|m| m.modified().ok());
if current_modified == last_modified {
continue;
}
...
}
```
`Credentials::from_sts` returns `AssumeRoleWithWebIdentity` credentials, which expire in **about an hour**. But the projected service account token is only rotated by the kubelet near **its own** expiry, and on EKS the default (`eks.amazonaws.com/token-expiration`) is **86400s**, rotated at roughly 80% of that.
So for ~19 hours the token file is unchanged, the loop `continue`s past every opportunity to refresh, and the credentials it derived at startup are already dead. The effect is **exactly one working hour of uploads per pod lifetime**, then every upload fails until the pod restarts.
### Observed
On a cubestore router using IRSA:
- uploads succeed for ~1 hour after each pod start, then fail continuously with `ExpiredToken`, at the cachestore upload cadence:
```
ERROR [cubestore::util] Error during Cachestore upload: CubeError { message: "AWS S3 error:
Got HTTP 400 with content 'ExpiredToken
The provided token has expired...."
```
- four separate restarts over a month each produced one working hour and nothing after it
- the projected volume showed `expirationSeconds: 86400`, and the loop was running (the 30s web-identity poll interval is applied) — it just never passed the mtime check
Consequences beyond failed uploads: remote snapshot pruning cannot advance, because `delete_old_snapshots()` only deletes what was successfully uploaded, and the remote metastore copy goes stale indefinitely.
### Workaround
Shorten the projected token's lifetime so the kubelet rotates it often enough to drive the refresh:
```yaml
# ServiceAccount annotation; 600 is the EKS minimum
eks.amazonaws.com/token-expiration: "600"
```
The kubelet then rewrites the file every ~480s, the mtime changes, and the loop re-derives. This fully resolves it for us (zero `ExpiredToken` since), at the cost of ~180 extra STS calls per day. It requires recreating pods, since the expiry is baked into the projected volume at admission.
That works, but it means correct behaviour depends on a cluster-side annotation that has no obvious connection to cubestore, and the default configuration is broken in a way that looks like partial support: it works just long enough to pass a smoke test.
### Suggested direction
The mtime check is a reasonable fast path, but it should not be the only trigger. Options, roughly in order of preference:
1. Track the credentials' own expiry and refresh when they are close to it, independent of the token file.
2. Refresh on a timer in web identity mode as well, with an interval well under the STS session length (the token file remains valid, so re-calling `from_sts` with the same JWT is enough).
3. At minimum, document that web identity mode requires a short `token-expiration`, since nothing in the current behaviour hints at it.
Happy to send a PR for whichever shape you prefer — I did not want to pick the refresh policy on your behalf.
### Not the same as #6795
[#6795](https://github.com/cube-js/cube/issues/6795) states that assuming an IAM role is not implemented. That reads as stale: the `CUBESTORE_AWS_WEB_IDENTITY_TOKEN_FILE` / `CUBESTORE_AWS_ROLE_ARN` path is honoured and does work, for one hour. Support is present but the refresh is gated, which is a narrower and more confusing failure than no support at all. I could not find an existing issue covering the refresh gap specifically.
Version: `cubejs/cubestore:v1.6.57`, and the code above is unchanged on `master`.
Contributor guide
Assessment
This issue has not been assessed yet.