cube-js / cube-js/cube

Cubestore: S3 refresh loop not detecting new AIM Role token from EKS IRSA file

Open
#11,248 2 comments 0 reactions 0 assignees View on GitHub
cube store
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

## Describe the bug

CubeStore fails to reliably refresh AWS S3 credentials when using AWS Web Identity credentials with a Kubernetes projected ServiceAccount token (for example, EKS IRSA).

CubeStore detects Web Identity token rotation by comparing the token file's `modified` timestamp (`mtime`):

```rust
let current_modified = std::fs::metadata(file)
.ok()
.and_then(|m| m.modified().ok());

if current_modified == last_modified {
continue;
}
```

However, Kubernetes projected ServiceAccount tokens are managed and rotated by kubelet. Relying only on the file modification timestamp is not a reliable way to detect that the JWT token content has changed.

As a result, CubeStore may fail to detect a rotated ServiceAccount token and does not call `Credentials::from_sts(...)` with the new token. The previously issued temporary AWS credentials eventually expire, causing S3 requests to fail with expired token/credentials errors.

Additionally, `last_modified` is updated before the STS credential refresh succeeds:

```rust
last_modified = current_modified;
info!("Web identity token file changed, refreshing S3 credentials");
```

If the STS request fails after `last_modified` has been updated, CubeStore considers the token file already processed and does not retry the refresh unless the file modification timestamp changes again.

## To Reproduce

1. Run CubeStore in Kubernetes/EKS using a ServiceAccount associated with an AWS IAM role via IRSA/Web Identity.

2. Configure CubeStore to use the projected ServiceAccount token:

```text
CUBESTORE_AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
CUBESTORE_AWS_WEB_IDENTITY_ROLE_ARN=arn:aws:iam:::role/
```

3. Do not configure static AWS access and secret keys.

4. Configure CubeStore to use an S3 bucket.

5. Start CubeStore and verify that initial S3 access works successfully.

6. Keep CubeStore running long enough for the Kubernetes projected ServiceAccount token and AWS STS credentials to rotate/expire.

7. Observe that CubeStore may continue using expired AWS credentials or fail to refresh credentials successfully.

8. S3 operations start failing with an expired token/credentials error.

Restarting the CubeStore pod restores S3 access because CubeStore reads the current projected ServiceAccount token and obtains new STS credentials during startup.

## Expected behavior

CubeStore should reliably refresh AWS credentials when using Kubernetes/EKS Web Identity authentication.

The refresh logic should not rely exclusively on the projected token file's `mtime`.

CubeStore should periodically read the token file and detect changes in the JWT content, for example by comparing the token content or a hash of the token.

The token should only be marked as successfully processed after `Credentials::from_sts(...)` succeeds.

If the STS credential refresh fails, CubeStore should retry the refresh on the next polling interval instead of waiting for another token file modification.

Expected behavior:

1. Kubernetes rotates the projected ServiceAccount token.
2. CubeStore detects that the JWT content has changed.
3. CubeStore calls `AssumeRoleWithWebIdentity` using the new JWT.
4. New temporary AWS credentials are obtained.
5. The S3 client/bucket credentials are replaced.
6. S3 access continues without interruption.

## Version:

1.6.69

## Additional context

The issue occurs when running CubeStore on Kubernetes/EKS with AWS IRSA and a projected ServiceAccount token.

The relevant credential refresh logic uses file metadata to detect token rotation:

```rust
let mut last_modified = token_file
.as_ref()
.and_then(|f| std::fs::metadata(f).ok()?.modified().ok());
```

and later:

```rust
let current_modified = std::fs::metadata(file)
.ok()
.and_then(|m| m.modified().ok());

if current_modified == last_modified {
continue;
}

last_modified = current_modified;
```

A more reliable implementation would periodically read the projected token and compare its content or hash:

```rust
let jwt = std::fs::read_to_string(file)?;

if last_token.as_deref() == Some(jwt.as_str()) {
continue;
}

let credentials = Credentials::from_sts(
arn,
"cubestore",
&jwt,
)?;

// Mark the token as processed only after STS succeeds.
last_token = Some(jwt);
```

Alternatively, CubeStore could refresh temporary AWS credentials based on their expiration time with an appropriate safety margin.

Currently, restarting the CubeStore pod is a workaround because the current ServiceAccount token is read and fresh STS credentials are obtained during startup.

Contributor guide

Open the contributing guide

Research direction

Locate CubeStore's Rust Web Identity/S3 credential refresh loop and the token-file polling logic shown in the issue. Start by tracing how the token is read, how last_modified is tracked, and how Credentials::from_sts(...) failures are handled. Done means rotated JWT content reliably triggers refreshed credentials, failed refreshes are retried, and S3 access continues after token rotation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, rust
Domain
authentication, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.