apache / apache/iceberg-rust

Support refreshing vended storage credentials for REST catalog tables

Open
#2,931 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

REST catalogs can vend short-lived storage credentials for accessing table data. Iceberg Java supports refreshing these credentials through cloud-specific refresh endpoints before they expire.

`iceberg-rust` does not currently refresh per-prefix credentials. Consequently, long-running reads, writes, and maintenance operations can fail once the initial S3 or GCS credentials expire.

Refresh support must preserve per-prefix credential isolation, table-scoped catalog authentication, and support tables referencing multiple storage prefixes or clouds.

This issue tracks the work required across the core storage API, OpenDAL backends, and REST catalog. The initial reference implementation was developed and reviewed in #2932.

### Describe the solution you'd like

Add refreshable storage credential support across the REST catalog, FileIO, and storage backends:

- Introduce a backend-independent credential-provider interface for FileIO
- Use the standard AWS and GCS refresh endpoint properties returned by REST catalogs
- Cache credentials per cloud and select the longest matching storage prefix

### Tasks

- [ ] **Security and independent fixes**
- Redact secret-bearing catalog and storage configuration.

- [ ] **Core credential-provider API** https://github.com/apache/iceberg-rust/pull/2976

- [ ] **OpenDAL S3 support and shared dynamic-credential machinery**
- Adapt credentials into reqsign’s AWS provider.

- [ ] **OpenDAL GCS support**
- Adapt OAuth2 credentials into reqsign’s Google provider.

- [ ] **REST credential-refresh provider**
- Fetch and parse credentials from catalog-provided endpoints.
- Cache credentials independently by cloud.
- Select the longest matching prefix.
- Implement cloud-compatible prefetch timing, jitter, and failure backoff.

- [ ] **REST catalog wiring**
- Construct the provider from table properties and attach it to table `FileIO`.

### Follow-up

Azure/ADLS refresh is not part of the initial S3 and GCS implementation. The remaining upstream OpenDAL capability is tracked by apache/opendal#8030. Azure support can follow once that API is merged and released.

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reference implementation in #2932 and the core credential-provider API in PR #2976. Then trace the REST catalog, FileIO, and OpenDAL S3/GCS areas named in the issue. Done means refreshable, per-cloud and per-prefix credentials are wired through table FileIO with secure redaction and the stated prefetch, jitter, and backoff behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, gcp, rust
Domain
backend-api-design, cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.