[Bug report] Web identity credentials are not applied to server-side Iceberg S3FileIO
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe what's wrong
The `aws-irsa` credential provider can read a WebIdentity token through `FileWebIdentityTokenSource`, but the server-side Iceberg paths do not consistently apply the vended storage credentials to Iceberg backend/FileIO configuration.
As a result, Iceberg `S3FileIO` may fall back to the AWS default credential chain when creating/loading table metadata, and the workflow only succeeds if the Gravitino process is also given `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` manually.
The expected behavior is that `credential-providers=aws-irsa` plus `s3-web-identity-token-source=file` is enough for Gravitino to fetch temporary credentials and use them in:
- Iceberg REST server backend FileIO
- Lakehouse Iceberg catalog backend FileIO
- Fileset credential vending
- GVFS access through fileset credentials
### Error message and/or stacktrace
`S3FileIO` can fail with:
`Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(...)`
The failing chain includes system properties, environment variables, AWS web identity token file env, profile, container credentials, and instance profile. This indicates the vended `AwsIrsaCredential` was not injected into the server-side Iceberg FileIO config.
### How to reproduce
1. Configure an S3/Iceberg catalog with:
- `credential-providers=aws-irsa`
- `s3-web-identity-token-source=file`
- `s3-web-identity-token-file=`
- `s3-role-arn=`
- `io-impl=org.apache.iceberg.aws.s3.S3FileIO`
- `warehouse=s3:///`
2. Do not set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or `AWS_SESSION_TOKEN`.
3. Create/load an Iceberg table through Gravitino/Iceberg REST.
4. The server-side FileIO may use AWS default credentials instead of the vended `AwsIrsaCredential`.
### Additional context
The existing file-based WebIdentity token source has unit coverage, but it should also have end-to-end coverage proving the whole chain works through Iceberg, Fileset, and GVFS.
Contributor guide
Assessment
This issue has not been assessed yet.