apache / apache/iceberg

Credential refresh for stage-created tables

Open
#13,554 3 comments 2 reactions 0 assignees View on GitHub
improvement not-stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

I'd like REST clients that are atomically creating a table to be able to refresh vended credentials when writing the table's files.

Currently, credential refresh works when the table exists, but when it does not, a simple e.g. Spark workflow like

```scala
df
.(...)
.writeTo(table)
.create()
```

does not refresh credentials.

IIUC, the atomic creation of a table works by the client first sending a `stageCreate` request to the REST server, using the returned `LoadTableResponse` for writing files, and finally commit-creating the table via an `updateTable` request to the server.

If the REST catalog provides the refresh properties within the `stageCreate` `LoadTableResponse` that includes the `loadCredentials` refresh endpoint **(1)**, and it abides by the REST specification to [throw a 404](https://github.com/apache/iceberg/blob/bcf9c69c098b54d31cbd803d62a2609d3814c3df/open-api/rest-catalog-open-api.yaml#L1205-L1207) when the table does not exist, then the Spark code above will throw because the table does not exist when refresh is attempted. Specifically, once the initial credentials within the `LoadTableResponse` have expired, the user will receive:

```
org.apache.iceberg.exceptions.RESTException: Unable to process: Table does not exist: ...
```

I suspected that I might have misunderstood the REST spec, but it looks like [Polaris](https://github.com/apache/polaris/blob/c43c546a227a3eddd0ed0d519f53d4b347c439a7/service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java#L590-L596) has the same interpretation, delegating to `loadTable`, which indeed returns a 404 if the table does not exist.

I think we should be able to support this common use case. Apologies if this is ongoing work, but I didn't find anything when searching online. I believe that this is achievable because `stageCreate` anyway returns credentials for a table that doesn't exist yet according to the catalog, which is what refresh requires for this use case.

(1) Importantly, I realise that a REST catalog could provide an endpoint different to the `loadCredentials` one and implement its own behaviour to handle the case described. However, my impression was that `loadCredentials` was designed to be the refresh endpoint, and I think there's value in having a specification for what a REST catalog should implement.

Curious for the community's thoughts. Happy to propose / contribute. cc @nastra

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.