apache / apache/polaris

table-default.* catalog properties not merged into S3FileIO for staged creates and doCommit

Open
#4,062 1 comment 0 reactions 0 assignees View on GitHub
stale
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

## Bug Report

**What version of Apache Polaris are you using?**
1.3.0-incubating

**What OS and processor architecture?**
Linux x86_64 (OpenShift / Kubernetes pod)

**What did you do?**
Configured Polaris with `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION=true` and `table-default.s3.endpoint` / `table-default.s3.path-style-access` catalog properties (required for S3-compatible stores like VAST that don't support STS AssumeRole). Then used DuckDB with `stage-create: true` to create an Iceberg table.

**What did you expect to see?**
S3FileIO uses `s3.endpoint` and `s3.path-style-access` from catalog properties when writing table metadata to S3.

**What did you see instead?**
S3FileIO ignores the catalog defaults and falls back to the default AWS S3 endpoint, causing 403 errors when the Polaris S3 credentials are not valid for real AWS.

---

## Root Cause

Three code paths fail to merge `table-default.*` catalog properties into the properties passed to S3FileIO:

### 1. `PolarisIcebergCatalogTableBuilder` constructor (`IcebergCatalog.java`)

`PolarisIcebergCatalogViewBuilder` correctly calls `withProperties(propertiesWithPrefix(..., "table-default."))` in its constructor. The table builder is missing this call.

```java
// Fix: add to PolarisIcebergCatalogTableBuilder constructor
withProperties(
PropertyUtil.propertiesWithPrefix(IcebergCatalog.this.properties(), "table-default."));
```

### 2. `IcebergCatalogHandler.stageTableCreateHelper()`

Clients like DuckDB send `stage-create: true`, which calls `TableMetadata.newTableMetadata()` directly, bypassing `buildTable()`. The `properties` map never receives catalog defaults.

```java
// Fix: merge catalog defaults before calling newTableMetadata()
Map mergedProperties = Maps.newHashMap();
if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
mergedProperties.putAll(PropertyUtil.propertiesWithPrefix(
polarisCatalog.properties(), "table-default."));
}
mergedProperties.putAll(properties);
```

### 3. `BasePolarisTableOperations.doCommit()` (`IcebergCatalog.java`)

DuckDB's staged create commits send sparse payloads with no `SetProperties` update, so `metadata.properties()` is empty of catalog defaults. The FileIO for the commit write is initialized only from `metadata.properties()`.

```java
// Fix: use tableDefaultProperties as baseline in doCommit()
Map commitFileIOProps = new HashMap<>(tableDefaultProperties);
commitFileIOProps.putAll(metadata.properties());
tableFileIO = loadFileIOForTableLike(..., commitFileIOProps, ...);
```

All three fixes are needed for DuckDB (staged create). Fix 1 alone is sufficient for direct creates (`stage-create: false`).

Contributor guide

Open the contributing guide

Research direction

Read the PolarisIcebergCatalogTableBuilder constructor in IcebergCatalog.java, then trace IcebergCatalogHandler.stageTableCreateHelper() and BasePolarisTableOperations.doCommit(). Reproduce the DuckDB staged-create path with the stated table-default.s3 settings and verify that direct and staged creates pass those defaults to S3FileIO and use the configured endpoint and path-style access.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.