Cubestore: S3 refresh loop not detecting new AIM Role token from EKS IRSA file
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
## Describe the bug
CubeStore fails to reliably refresh AWS S3 credentials when using AWS Web Identity credentials with a Kubernetes projected ServiceAccount token (for example, EKS IRSA).
CubeStore detects Web Identity token rotation by comparing the token file's `modified` timestamp (`mtime`):
```rust
let current_modified = std::fs::metadata(file)
.ok()
.and_then(|m| m.modified().ok());
if current_modified == last_modified {
continue;
}
```
However, Kubernetes projected ServiceAccount tokens are managed and rotated by kubelet. Relying only on the file modification timestamp is not a reliable way to detect that the JWT token content has changed.
As a result, CubeStore may fail to detect a rotated ServiceAccount token and does not call `Credentials::from_sts(...)` with the new token. The previously issued temporary AWS credentials eventually expire, causing S3 requests to fail with expired token/credentials errors.
Additionally, `last_modified` is updated before the STS credential refresh succeeds:
```rust
last_modified = current_modified;
info!("Web identity token file changed, refreshing S3 credentials");
```
If the STS request fails after `last_modified` has been updated, CubeStore considers the token file already processed and does not retry the refresh unless the file modification timestamp changes again.
## To Reproduce
1. Run CubeStore in Kubernetes/EKS using a ServiceAccount associated with an AWS IAM role via IRSA/Web Identity.
2. Configure CubeStore to use the projected ServiceAccount token:
```text
CUBESTORE_AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
CUBESTORE_AWS_WEB_IDENTITY_ROLE_ARN=arn:aws:iam:::role/
```
3. Do not configure static AWS access and secret keys.
4. Configure CubeStore to use an S3 bucket.
5. Start CubeStore and verify that initial S3 access works successfully.
6. Keep CubeStore running long enough for the Kubernetes projected ServiceAccount token and AWS STS credentials to rotate/expire.
7. Observe that CubeStore may continue using expired AWS credentials or fail to refresh credentials successfully.
8. S3 operations start failing with an expired token/credentials error.
Restarting the CubeStore pod restores S3 access because CubeStore reads the current projected ServiceAccount token and obtains new STS credentials during startup.
## Expected behavior
CubeStore should reliably refresh AWS credentials when using Kubernetes/EKS Web Identity authentication.
The refresh logic should not rely exclusively on the projected token file's `mtime`.
CubeStore should periodically read the token file and detect changes in the JWT content, for example by comparing the token content or a hash of the token.
The token should only be marked as successfully processed after `Credentials::from_sts(...)` succeeds.
If the STS credential refresh fails, CubeStore should retry the refresh on the next polling interval instead of waiting for another token file modification.
Expected behavior:
1. Kubernetes rotates the projected ServiceAccount token.
2. CubeStore detects that the JWT content has changed.
3. CubeStore calls `AssumeRoleWithWebIdentity` using the new JWT.
4. New temporary AWS credentials are obtained.
5. The S3 client/bucket credentials are replaced.
6. S3 access continues without interruption.
## Version:
1.6.69
## Additional context
The issue occurs when running CubeStore on Kubernetes/EKS with AWS IRSA and a projected ServiceAccount token.
The relevant credential refresh logic uses file metadata to detect token rotation:
```rust
let mut last_modified = token_file
.as_ref()
.and_then(|f| std::fs::metadata(f).ok()?.modified().ok());
```
and later:
```rust
let current_modified = std::fs::metadata(file)
.ok()
.and_then(|m| m.modified().ok());
if current_modified == last_modified {
continue;
}
last_modified = current_modified;
```
A more reliable implementation would periodically read the projected token and compare its content or hash:
```rust
let jwt = std::fs::read_to_string(file)?;
if last_token.as_deref() == Some(jwt.as_str()) {
continue;
}
let credentials = Credentials::from_sts(
arn,
"cubestore",
&jwt,
)?;
// Mark the token as processed only after STS succeeds.
last_token = Some(jwt);
```
Alternatively, CubeStore could refresh temporary AWS credentials based on their expiration time with an appropriate safety margin.
Currently, restarting the CubeStore pod is a workaround because the current ServiceAccount token is read and fresh STS credentials are obtained during startup.
Contributor guide
Research direction
Locate CubeStore's Rust Web Identity/S3 credential refresh loop and the token-file polling logic shown in the issue. Start by tracing how the token is read, how last_modified is tracked, and how Credentials::from_sts(...) failures are handled. Done means rotated JWT content reliably triggers refreshed credentials, failed refreshes are retried, and S3 access continues after token rotation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, rust
- Domain
- authentication, backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100