[Bug] S3FileIO.refreshStorageCredentials() fails with "Invalid credentials endpoint: null" when using REST catalog with vended credentials
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
## Apache Iceberg version
1.11.0
## Query engine
Spark 4.0
## Please describe the bug 🐞
When using a REST catalog (Apache Polaris) with `X-Iceberg-Access-Delegation: vended-credentials`, the new `S3FileIO.refreshStorageCredentials()` method introduced in PR #15678 fails with:
```
WARN S3FileIO: Failed to refresh storage credentials
java.lang.IllegalArgumentException: Invalid credentials endpoint: null
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.iceberg.aws.s3.VendedCredentialsProvider.(VendedCredentialsProvider.java:61)
at org.apache.iceberg.aws.s3.VendedCredentialsProvider.create(VendedCredentialsProvider.java:88)
at org.apache.iceberg.aws.s3.S3FileIO.refreshStorageCredentials(S3FileIO.java:464)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
```
## Root Cause Analysis
`S3FileIO.refreshStorageCredentials()` calls `VendedCredentialsProvider.create(properties)` passing the raw catalog properties from the `SerializableMap`.
`VendedCredentialsProvider` expects the property `credentials.uri` (constant `VendedCredentialsProvider.URI`) to be present in the map. However, the raw catalog properties only contain:
- `uri` — the catalog base endpoint
- `client.refresh-credentials-endpoint` — the relative path returned by the REST catalog server (e.g. `v1/{warehouse}/namespaces/{ns}/tables/{table}/credentials`)
The resolution of `uri` + `client.refresh-credentials-endpoint` → `credentials.uri` is normally done by `RESTSessionCatalog` when it constructs the properties for `VendedCredentialsProvider` used by the internal S3 client (`PrefixedS3Client`). But in the `refreshStorageCredentials()` code path, this resolution is skipped — the raw properties are passed directly.
### Evidence from bytecode decompilation
```
// VendedCredentialsProvider constructor:
ldc #35 // String "credentials.uri"
invokeinterface Map.get
...
ldc #72 // String "Invalid credentials endpoint: null"
invokestatic Preconditions.checkArgument
```
### Polaris loadTable response (with X-Iceberg-Access-Delegation: vended-credentials)
```json
{
"s3.access-key-id": "ASIA...",
"s3.secret-access-key": "...",
"s3.session-token": "...",
"client.refresh-credentials-endpoint": "v1/products_warehouse/namespaces/gold_crd/tables/dim_contract/credentials",
"expiration-time": "1787596581000",
"s3.session-token-expires-at-ms": "1787596581000",
"client.region": "eu-west-1"
}
```
The server correctly provides `client.refresh-credentials-endpoint`, but the client does not resolve it into `credentials.uri` before passing it to `VendedCredentialsProvider`.
### Comparison with Iceberg 1.10.0
In 1.10.0, `S3FileIO` does NOT have `refreshStorageCredentials()`. The `VendedCredentialsProvider` is only used internally by `PrefixedS3Client` where `credentials.uri` is correctly constructed by the `RESTSessionCatalog`. This is a regression introduced in 1.11.0 by PR #15678.
## To Reproduce
1. Set up a REST catalog server (e.g., Apache Polaris) with S3 storage and vended credentials enabled
2. Configure Spark with:
```properties
spark.sql.catalog.mycatalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.mycatalog.type=rest
spark.sql.catalog.mycatalog.uri=http:///api/catalog
spark.sql.catalog.mycatalog.header.X-Iceberg-Access-Delegation=vended-credentials
spark.sql.catalog.mycatalog.credential=:
```
3. Load a table and wait for the credential refresh scheduler to trigger (~5 minutes before credential expiration)
4. Observe the WARN in logs
## Expected behavior
`S3FileIO.refreshStorageCredentials()` should either:
1. Resolve `uri` + `client.refresh-credentials-endpoint` into `credentials.uri` before calling `VendedCredentialsProvider.create()`
2. Or receive pre-resolved properties from `RESTSessionCatalog`
## Actual behavior
The method passes raw catalog properties to `VendedCredentialsProvider.create()`, which fails because `credentials.uri` is absent.
## Current workaround
The warning is non-blocking as long as `cache.expiration-interval-ms` (default 30s) causes frequent `loadTable` calls that provide fresh credentials before expiration. No data loss occurs. However, for long-running single writes (>1h), the credentials could expire without being refreshed.
## Environment
- Iceberg: 1.11.0 (iceberg-spark-runtime-4.0_2.13-1.11.0.jar)
- Spark: 4.0
- REST Catalog: Apache Polaris 1.4.1 (also verified against Polaris 1.7.0 docs — same behavior)
- Storage: AWS S3
Contributor guide
Research direction
Start with S3FileIO.refreshStorageCredentials() and trace how its raw SerializableMap reaches VendedCredentialsProvider.create(); compare that path with RESTSessionCatalog's construction of the properties used by PrefixedS3Client. The fix is complete when the refresh path resolves the REST endpoint into credentials.uri and vended credentials refresh without the Invalid credentials endpoint warning in the reported REST catalog setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100