Credential refresh for stage-created tables
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
I'd like REST clients that are atomically creating a table to be able to refresh vended credentials when writing the table's files.
Currently, credential refresh works when the table exists, but when it does not, a simple e.g. Spark workflow like
```scala
df
.(...)
.writeTo(table)
.create()
```
does not refresh credentials.
IIUC, the atomic creation of a table works by the client first sending a `stageCreate` request to the REST server, using the returned `LoadTableResponse` for writing files, and finally commit-creating the table via an `updateTable` request to the server.
If the REST catalog provides the refresh properties within the `stageCreate` `LoadTableResponse` that includes the `loadCredentials` refresh endpoint **(1)**, and it abides by the REST specification to [throw a 404](https://github.com/apache/iceberg/blob/bcf9c69c098b54d31cbd803d62a2609d3814c3df/open-api/rest-catalog-open-api.yaml#L1205-L1207) when the table does not exist, then the Spark code above will throw because the table does not exist when refresh is attempted. Specifically, once the initial credentials within the `LoadTableResponse` have expired, the user will receive:
```
org.apache.iceberg.exceptions.RESTException: Unable to process: Table does not exist: ...
```
I suspected that I might have misunderstood the REST spec, but it looks like [Polaris](https://github.com/apache/polaris/blob/c43c546a227a3eddd0ed0d519f53d4b347c439a7/service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java#L590-L596) has the same interpretation, delegating to `loadTable`, which indeed returns a 404 if the table does not exist.
I think we should be able to support this common use case. Apologies if this is ongoing work, but I didn't find anything when searching online. I believe that this is achievable because `stageCreate` anyway returns credentials for a table that doesn't exist yet according to the catalog, which is what refresh requires for this use case.
(1) Importantly, I realise that a REST catalog could provide an endpoint different to the `loadCredentials` one and implement its own behaviour to handle the case described. However, my impression was that `loadCredentials` was designed to be the refresh endpoint, and I think there's value in having a specification for what a REST catalog should implement.
Curious for the community's thoughts. Happy to propose / contribute. cc @nastra
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Assessment
This issue has not been assessed yet.