table-default.* catalog properties not merged into S3FileIO for staged creates and doCommit
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
## Bug Report
**What version of Apache Polaris are you using?**
1.3.0-incubating
**What OS and processor architecture?**
Linux x86_64 (OpenShift / Kubernetes pod)
**What did you do?**
Configured Polaris with `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION=true` and `table-default.s3.endpoint` / `table-default.s3.path-style-access` catalog properties (required for S3-compatible stores like VAST that don't support STS AssumeRole). Then used DuckDB with `stage-create: true` to create an Iceberg table.
**What did you expect to see?**
S3FileIO uses `s3.endpoint` and `s3.path-style-access` from catalog properties when writing table metadata to S3.
**What did you see instead?**
S3FileIO ignores the catalog defaults and falls back to the default AWS S3 endpoint, causing 403 errors when the Polaris S3 credentials are not valid for real AWS.
---
## Root Cause
Three code paths fail to merge `table-default.*` catalog properties into the properties passed to S3FileIO:
### 1. `PolarisIcebergCatalogTableBuilder` constructor (`IcebergCatalog.java`)
`PolarisIcebergCatalogViewBuilder` correctly calls `withProperties(propertiesWithPrefix(..., "table-default."))` in its constructor. The table builder is missing this call.
```java
// Fix: add to PolarisIcebergCatalogTableBuilder constructor
withProperties(
PropertyUtil.propertiesWithPrefix(IcebergCatalog.this.properties(), "table-default."));
```
### 2. `IcebergCatalogHandler.stageTableCreateHelper()`
Clients like DuckDB send `stage-create: true`, which calls `TableMetadata.newTableMetadata()` directly, bypassing `buildTable()`. The `properties` map never receives catalog defaults.
```java
// Fix: merge catalog defaults before calling newTableMetadata()
Map mergedProperties = Maps.newHashMap();
if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
mergedProperties.putAll(PropertyUtil.propertiesWithPrefix(
polarisCatalog.properties(), "table-default."));
}
mergedProperties.putAll(properties);
```
### 3. `BasePolarisTableOperations.doCommit()` (`IcebergCatalog.java`)
DuckDB's staged create commits send sparse payloads with no `SetProperties` update, so `metadata.properties()` is empty of catalog defaults. The FileIO for the commit write is initialized only from `metadata.properties()`.
```java
// Fix: use tableDefaultProperties as baseline in doCommit()
Map commitFileIOProps = new HashMap<>(tableDefaultProperties);
commitFileIOProps.putAll(metadata.properties());
tableFileIO = loadFileIOForTableLike(..., commitFileIOProps, ...);
```
All three fixes are needed for DuckDB (staged create). Fix 1 alone is sufficient for direct creates (`stage-create: false`).
Contributor guide
Research direction
Read the PolarisIcebergCatalogTableBuilder constructor in IcebergCatalog.java, then trace IcebergCatalogHandler.stageTableCreateHelper() and BasePolarisTableOperations.doCommit(). Reproduce the DuckDB staged-create path with the stated table-default.s3 settings and verify that direct and staged creates pass those defaults to S3FileIO and use the configured endpoint and path-style access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100