AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
We're running Polaris on EKS with IRSA, using the Kubernetes service account role as the Polaris AWS identity.
While investigating STS usage, we noticed that `AssumeRoleWithWebIdentity` calls for the Polaris IRSA role appear to track SigV4 connection-credential `AssumeRole` calls almost one-for-one.
The downstream SigV4 `AssumeRole` calls may be legitimate catalog traffic. The part that looks suspicious is that Polaris appears to resolve its own base WebIdentity credentials repeatedly instead of reusing the AWS SDK credential provider/cache across those calls.
### To Reproduce
1. Run Polaris on EKS with IRSA.
2. Do not configure static AWS credentials for Polaris.
3. Configure a federated Glue catalog using SigV4 authentication.
4. Send normal catalog traffic to the federated catalog.
5. Query CloudTrail for both:
- `AssumeRoleWithWebIdentity` on the Polaris service account role
- `AssumeRole` on the same role used by the SigV4 connection configuration
I don't have a small local reproducer yet; the observations below are from EKS deployments.
### Actual Behavior
In one deployment, there was one running Polaris pod with no restarts:
```text
Polaris deployment replicas: 1
Polaris pod restarts: 0
Polaris image: apache/polaris:1.3.0-incubating
```
For the same one-minute CloudTrail window, we saw:
```text
Window: 2026-06-11T18:19:00Z to 2026-06-11T18:20:00Z
AssumeRoleWithWebIdentity events for the Polaris IRSA role:
- Total events: 62
- aws-sdk-java/2.39.2 ... api/STS#2.39.x ... lang/java#21.0.9: 60
- Other AWS SDK clients: 2
- Java events came from one source IP
- Java events had distinct aws-sdk-java- role session names
AssumeRole events for the SigV4 role in the same window:
- Total events: 60
- aws-sdk-java/2.39.2 ... api/STS#2.39.x ... lang/java#21.0.9: 60
- Role session name: polaris
```
A second deployment showed the same general pattern at higher volume:
```text
Window: 2026-06-11T18:19:00Z to 2026-06-11T18:20:00Z
AssumeRoleWithWebIdentity events for the Polaris IRSA role:
- Total events: 632
- aws-sdk-java/2.39.2 ... api/STS#2.39.x ... lang/java#21.0.9: 632
- Events came from one source IP
- Events had distinct aws-sdk-java- role session names
AssumeRole events for the SigV4 role in the same window:
- Total events: 1,281
- Role session name "polaris": 1,279
- Role session name "PolarisAwsCredentialsStorageIntegration": 2
```
Sanitized `AssumeRoleWithWebIdentity` event shape:
```json
{
"eventTime": "2026-06-11T18:19:00Z",
"eventName": "AssumeRoleWithWebIdentity",
"userAgent": "aws-sdk-java/2.39.2 ... api/STS#2.39.x ... lang/java#21.0.9",
"roleArn": "arn:aws:iam:::role/",
"roleSessionName": "aws-sdk-java-",
"sourceIPAddress": ""
}
```
### Expected Behavior
Polaris may legitimately need to call downstream STS `AssumeRole` for SigV4 connection credentials.
However, those downstream `AssumeRole` calls should not require Polaris to resolve its own base IRSA WebIdentity credentials every time. I would expect the base AWS SDK credentials provider to be reused, so `AssumeRoleWithWebIdentity` happens according to the normal credential refresh cadence rather than tracking catalog requests or SigV4 `AssumeRole` calls one-for-one.
### Additional context
One possible area to check is the service identity path. In 1.3.0-incubating, `AwsIamServiceIdentityConfiguration.awsCredentialsProvider()` builds a `DefaultCredentialsProvider` when static credentials are not configured:
```java
return DefaultCredentialsProvider.builder().build();
```
That is reached through request-scoped service identity / SigV4 connection credential code. If that path creates a new AWS credentials provider for each request, the AWS SDK's WebIdentity credential cache may not be reused across downstream `AssumeRole` calls.
I checked current `main` and this part of the code still appears to use `DefaultCredentialsProvider.builder().build()` in the same service identity path.
### System information
OS: Linux on EKS
Java: 21
Polaris version observed: 1.3.0-incubating
AWS SDK user agent observed: aws-sdk-java/2.39.2
Storage: AWS S3
Federated catalog: AWS Glue Iceberg REST endpoint with SigV4 authentication
Auth setup: EKS IRSA, no static AWS access keys configured for Polaris
Contributor guide
Research direction
Start with AwsIamServiceIdentityConfiguration.awsCredentialsProvider() and trace the request-scoped service identity and SigV4 connection credential path around DefaultCredentialsProvider.builder().build(). Compare provider lifetimes with the EKS/IRSA CloudTrail observations. Done means confirming whether providers are recreated per request and establishing reuse so AssumeRoleWithWebIdentity follows normal refresh behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, kubernetes
- Domain
- authentication, backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100