apache / apache/iceberg

[Bug] S3FileIO.refreshStorageCredentials() fails with "Invalid credentials endpoint: null" when using REST catalog with vended credentials

Open
#17,810 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

## Apache Iceberg version

1.11.0

## Query engine

Spark 4.0

## Please describe the bug 🐞

When using a REST catalog (Apache Polaris) with `X-Iceberg-Access-Delegation: vended-credentials`, the new `S3FileIO.refreshStorageCredentials()` method introduced in PR #15678 fails with:

```
WARN S3FileIO: Failed to refresh storage credentials
java.lang.IllegalArgumentException: Invalid credentials endpoint: null
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.iceberg.aws.s3.VendedCredentialsProvider.(VendedCredentialsProvider.java:61)
at org.apache.iceberg.aws.s3.VendedCredentialsProvider.create(VendedCredentialsProvider.java:88)
at org.apache.iceberg.aws.s3.S3FileIO.refreshStorageCredentials(S3FileIO.java:464)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
```

## Root Cause Analysis

`S3FileIO.refreshStorageCredentials()` calls `VendedCredentialsProvider.create(properties)` passing the raw catalog properties from the `SerializableMap`.

`VendedCredentialsProvider` expects the property `credentials.uri` (constant `VendedCredentialsProvider.URI`) to be present in the map. However, the raw catalog properties only contain:
- `uri` — the catalog base endpoint
- `client.refresh-credentials-endpoint` — the relative path returned by the REST catalog server (e.g. `v1/{warehouse}/namespaces/{ns}/tables/{table}/credentials`)

The resolution of `uri` + `client.refresh-credentials-endpoint` → `credentials.uri` is normally done by `RESTSessionCatalog` when it constructs the properties for `VendedCredentialsProvider` used by the internal S3 client (`PrefixedS3Client`). But in the `refreshStorageCredentials()` code path, this resolution is skipped — the raw properties are passed directly.

### Evidence from bytecode decompilation

```
// VendedCredentialsProvider constructor:
ldc #35 // String "credentials.uri"
invokeinterface Map.get
...
ldc #72 // String "Invalid credentials endpoint: null"
invokestatic Preconditions.checkArgument
```

### Polaris loadTable response (with X-Iceberg-Access-Delegation: vended-credentials)

```json
{
"s3.access-key-id": "ASIA...",
"s3.secret-access-key": "...",
"s3.session-token": "...",
"client.refresh-credentials-endpoint": "v1/products_warehouse/namespaces/gold_crd/tables/dim_contract/credentials",
"expiration-time": "1787596581000",
"s3.session-token-expires-at-ms": "1787596581000",
"client.region": "eu-west-1"
}
```

The server correctly provides `client.refresh-credentials-endpoint`, but the client does not resolve it into `credentials.uri` before passing it to `VendedCredentialsProvider`.

### Comparison with Iceberg 1.10.0

In 1.10.0, `S3FileIO` does NOT have `refreshStorageCredentials()`. The `VendedCredentialsProvider` is only used internally by `PrefixedS3Client` where `credentials.uri` is correctly constructed by the `RESTSessionCatalog`. This is a regression introduced in 1.11.0 by PR #15678.

## To Reproduce

1. Set up a REST catalog server (e.g., Apache Polaris) with S3 storage and vended credentials enabled
2. Configure Spark with:
```properties
spark.sql.catalog.mycatalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.mycatalog.type=rest
spark.sql.catalog.mycatalog.uri=http:///api/catalog
spark.sql.catalog.mycatalog.header.X-Iceberg-Access-Delegation=vended-credentials
spark.sql.catalog.mycatalog.credential=:
```
3. Load a table and wait for the credential refresh scheduler to trigger (~5 minutes before credential expiration)
4. Observe the WARN in logs

## Expected behavior

`S3FileIO.refreshStorageCredentials()` should either:
1. Resolve `uri` + `client.refresh-credentials-endpoint` into `credentials.uri` before calling `VendedCredentialsProvider.create()`
2. Or receive pre-resolved properties from `RESTSessionCatalog`

## Actual behavior

The method passes raw catalog properties to `VendedCredentialsProvider.create()`, which fails because `credentials.uri` is absent.

## Current workaround

The warning is non-blocking as long as `cache.expiration-interval-ms` (default 30s) causes frequent `loadTable` calls that provide fresh credentials before expiration. No data loss occurs. However, for long-running single writes (>1h), the credentials could expire without being refreshed.

## Environment

- Iceberg: 1.11.0 (iceberg-spark-runtime-4.0_2.13-1.11.0.jar)
- Spark: 4.0
- REST Catalog: Apache Polaris 1.4.1 (also verified against Polaris 1.7.0 docs — same behavior)
- Storage: AWS S3

Contributor guide

Open the contributing guide

Research direction

Start with S3FileIO.refreshStorageCredentials() and trace how its raw SerializableMap reaches VendedCredentialsProvider.create(); compare that path with RESTSessionCatalog's construction of the properties used by PrefixedS3Client. The fix is complete when the refresh path resolves the REST endpoint into credentials.uri and vended credentials refresh without the Invalid credentials endpoint warning in the reported REST catalog setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.