REST OAuth2Manager standalone tableSession drops custom `header.*` properties, breaking Polaris multi-tenant credential vending
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.1
### Query engine
Kafka Connect
### Please describe the bug 🐞
## Description
### Summary
Apache Iceberg Java 1.10.1 does not propagate configured REST catalog custom headers (`header.*`) into the standalone `OAuth2Manager.tableSession(RESTClient, Map)` path. This causes credential vending to fail when Apache Polaris is configured with `require-header=true` for multi-tenant realm routing.
The issue manifests when a long-running process (such as Kafka Connect Sink) holds a FileIO instance whose vended S3 credentials expire, triggering a credential refresh via the standalone table session path — which creates an HTTP session without the required custom headers.
### Environment
| Component | Version |
|-----------|---------|
| Apache Iceberg (Java) | 1.10.1 |
| Apache Iceberg Kafka Connect Sink | 1.10.1 |
| Apache Polaris | 1.4.1 |
| Auth | External OIDC (Keycloak) |
| Storage | S3 (AWS and MinIO for reproduction) |
| Credential vending | Polaris STS (`X-Iceberg-Access-Delegation: true`) |
### Configuration
```properties
iceberg.catalog.type=rest
iceberg.catalog.uri=http://polaris:8181/api/catalog
iceberg.catalog.security=OAUTH2
iceberg.catalog.oauth2-server-uri=http://keycloak/realms/iceberg/protocol/openid-connect/token
iceberg.catalog.credential=:
iceberg.catalog.token-exchange-enabled=false
iceberg.catalog.header.Polaris-Realm=
iceberg.catalog.header.X-Iceberg-Access-Delegation=true
iceberg.catalog.fs.native-s3.enabled=true
```
Polaris configuration:
```properties
polaris.realm-context.require-header=true
```
### Expected behavior
All HTTP calls to Polaris should include configured custom headers (`header.*`), including:
- Catalog metadata calls (`GET /v1/config`, `GET /v1/.../tables/...`)
- Credential vending / refresh calls triggered by the standalone FileIO path
- Any call made by `OAuth2Manager.tableSession(RESTClient, Map)`
### Actual behavior
When a FileIO needs to refresh its S3 credentials after they expire (e.g., after 900s with MinIO STS or 3600s with AWS STS), it invokes `OAuth2Manager.tableSession(RESTClient, Map properties)`. This method creates an `AuthSession` using only `OAuth2Util.authHeaders(config.token())`, which contains only the `Authorization` header — not the custom `header.*` properties (like `Polaris-Realm`).
The resulting HTTP call to Polaris is rejected with:
```
RESTException: Unable to process: Missing or invalid realm
```
The task crashes and enters a crash loop.
### Root cause
In `OAuth2Manager.java` (line ~200 in 1.10.1):
```java
public AuthSession tableSession(RESTClient sharedClient, Map properties) {
AuthConfig config = AuthConfig.fromProperties(properties);
Map headers = OAuth2Util.authHeaders(config.token());
// ↑ Only Authorization header, NOT the custom header.* properties
OAuth2Util.AuthSession parent = new OAuth2Util.AuthSession(headers, config);
...
}
```
The `properties` map passed to this method **does contain** the `header.Polaris-Realm` property, but it is never extracted via `RESTUtil.configHeaders(properties)` and merged into the session headers.
Additionally, `newSessionFromTokenResponse` calls:
```java
OAuth2Util.fetchToken(refreshClient, Map.of(), ...)
// ↑ empty headers map
```
This means even token refresh calls made through this path will not include custom headers.
### Reproduction
Minimal reproduction using Docker Compose (Kafka + Polaris + Keycloak + MinIO):
1. Configure Polaris with `require-header=true` and `STORAGE_CREDENTIAL_DURATION_SECONDS=900`
2. Configure Iceberg Kafka Connect Sink with:
- `header.Polaris-Realm=local`
- `X-Iceberg-Access-Delegation=true`
- `token-exchange-enabled=false`
- `fs.native-s3.enabled=true`
- `commit.interval-ms=1200000` (20 min, longer than credential duration)
3. Start the connector and produce a message
4. Wait >15 min for S3 credentials to expire
5. The next commit attempt triggers a credential refresh via the standalone `tableSession` path
6. Polaris rejects the request: `Missing or invalid realm`
7. Task crashes
Full reproduction environment: available on request.
### Who is affected
Any long-running Iceberg client using:
- REST catalog with custom `header.*` properties
- Polaris multi-tenant with `require-header=true`
- Credential vending (`X-Iceberg-Access-Delegation`)
- A FileIO that outlives the vended credential duration
This primarily affects **Kafka Connect Sink** (long-running, holds FileIO instances across commits) but could also affect long-running Spark jobs or any standalone component using `AuthManager.tableSession(RESTClient, Map)`.
### Suggested fix
In `OAuth2Manager.tableSession(RESTClient, Map)`, merge custom headers from properties into the session:
```java
public AuthSession tableSession(RESTClient sharedClient, Map properties) {
AuthConfig config = AuthConfig.fromProperties(properties);
// Fix: include both auth headers AND custom headers from properties
Map customHeaders = RESTUtil.configHeaders(properties);
Map authHeaders = OAuth2Util.authHeaders(config.token());
Map headers = RESTUtil.merge(customHeaders, authHeaders);
OAuth2Util.AuthSession parent = new OAuth2Util.AuthSession(headers, config);
...
}
```
Also ensure that `newSessionFromTokenResponse` passes the configured headers instead of `Map.of()`:
```java
OAuth2Util.fetchToken(refreshClient, configuredHeaders, credential, scope, oauth2ServerUri, ...);
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start in OAuth2Manager.java at tableSession(RESTClient, Map) and trace how its AuthSession headers reach newSessionFromTokenResponse and OAuth2Util.fetchToken. Check RESTUtil.configHeaders(properties) and the token-refresh path first. Done means configured header.* values accompany catalog, credential-vending, and refresh requests made through this standalone session path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- authentication, backend-api-design
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100