cube-js / cube-js/cube

cubestore: web identity S3 credentials are only refreshed when the token file changes, so they expire after ~1h

Open
#11,622 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

### Summary

`spawn_creds_refresh_loop` in `remotefs/s3.rs` does refresh S3 credentials, but in web identity mode it re-derives them **only when the token file's mtime changes**:

```rust
// In web identity mode, only refresh when the token file changed.
if let (Some(ref file), Some(_)) = (&token_file, &role_arn) {
let current_modified = std::fs::metadata(file).ok().and_then(|m| m.modified().ok());
if current_modified == last_modified {
continue;
}
...
}
```

`Credentials::from_sts` returns `AssumeRoleWithWebIdentity` credentials, which expire in **about an hour**. But the projected service account token is only rotated by the kubelet near **its own** expiry, and on EKS the default (`eks.amazonaws.com/token-expiration`) is **86400s**, rotated at roughly 80% of that.

So for ~19 hours the token file is unchanged, the loop `continue`s past every opportunity to refresh, and the credentials it derived at startup are already dead. The effect is **exactly one working hour of uploads per pod lifetime**, then every upload fails until the pod restarts.

### Observed

On a cubestore router using IRSA:

- uploads succeed for ~1 hour after each pod start, then fail continuously with `ExpiredToken`, at the cachestore upload cadence:

```
ERROR [cubestore::util] Error during Cachestore upload: CubeError { message: "AWS S3 error:
Got HTTP 400 with content 'ExpiredToken
The provided token has expired...."
```

- four separate restarts over a month each produced one working hour and nothing after it
- the projected volume showed `expirationSeconds: 86400`, and the loop was running (the 30s web-identity poll interval is applied) — it just never passed the mtime check

Consequences beyond failed uploads: remote snapshot pruning cannot advance, because `delete_old_snapshots()` only deletes what was successfully uploaded, and the remote metastore copy goes stale indefinitely.

### Workaround

Shorten the projected token's lifetime so the kubelet rotates it often enough to drive the refresh:

```yaml
# ServiceAccount annotation; 600 is the EKS minimum
eks.amazonaws.com/token-expiration: "600"
```

The kubelet then rewrites the file every ~480s, the mtime changes, and the loop re-derives. This fully resolves it for us (zero `ExpiredToken` since), at the cost of ~180 extra STS calls per day. It requires recreating pods, since the expiry is baked into the projected volume at admission.

That works, but it means correct behaviour depends on a cluster-side annotation that has no obvious connection to cubestore, and the default configuration is broken in a way that looks like partial support: it works just long enough to pass a smoke test.

### Suggested direction

The mtime check is a reasonable fast path, but it should not be the only trigger. Options, roughly in order of preference:

1. Track the credentials' own expiry and refresh when they are close to it, independent of the token file.
2. Refresh on a timer in web identity mode as well, with an interval well under the STS session length (the token file remains valid, so re-calling `from_sts` with the same JWT is enough).
3. At minimum, document that web identity mode requires a short `token-expiration`, since nothing in the current behaviour hints at it.

Happy to send a PR for whichever shape you prefer — I did not want to pick the refresh policy on your behalf.

### Not the same as #6795

[#6795](https://github.com/cube-js/cube/issues/6795) states that assuming an IAM role is not implemented. That reads as stale: the `CUBESTORE_AWS_WEB_IDENTITY_TOKEN_FILE` / `CUBESTORE_AWS_ROLE_ARN` path is honoured and does work, for one hour. Support is present but the refresh is gated, which is a narrower and more confusing failure than no support at all. I could not find an existing issue covering the refresh gap specifically.

Version: `cubejs/cubestore:v1.6.57`, and the code above is unchanged on `master`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.