Support refreshing vended storage credentials for REST catalog tables
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
REST catalogs can vend short-lived storage credentials for accessing table data. Iceberg Java supports refreshing these credentials through cloud-specific refresh endpoints before they expire.
`iceberg-rust` does not currently refresh per-prefix credentials. Consequently, long-running reads, writes, and maintenance operations can fail once the initial S3 or GCS credentials expire.
Refresh support must preserve per-prefix credential isolation, table-scoped catalog authentication, and support tables referencing multiple storage prefixes or clouds.
This issue tracks the work required across the core storage API, OpenDAL backends, and REST catalog. The initial reference implementation was developed and reviewed in #2932.
### Describe the solution you'd like
Add refreshable storage credential support across the REST catalog, FileIO, and storage backends:
- Introduce a backend-independent credential-provider interface for FileIO
- Use the standard AWS and GCS refresh endpoint properties returned by REST catalogs
- Cache credentials per cloud and select the longest matching storage prefix
### Tasks
- [ ] **Security and independent fixes**
- Redact secret-bearing catalog and storage configuration.
- [ ] **Core credential-provider API** https://github.com/apache/iceberg-rust/pull/2976
- [ ] **OpenDAL S3 support and shared dynamic-credential machinery**
- Adapt credentials into reqsign’s AWS provider.
- [ ] **OpenDAL GCS support**
- Adapt OAuth2 credentials into reqsign’s Google provider.
- [ ] **REST credential-refresh provider**
- Fetch and parse credentials from catalog-provided endpoints.
- Cache credentials independently by cloud.
- Select the longest matching prefix.
- Implement cloud-compatible prefetch timing, jitter, and failure backoff.
- [ ] **REST catalog wiring**
- Construct the provider from table properties and attach it to table `FileIO`.
### Follow-up
Azure/ADLS refresh is not part of the initial S3 and GCS implementation. The remaining upstream OpenDAL capability is tracked by apache/opendal#8030. Azure support can follow once that API is merged and released.
### Willingness to contribute
I can contribute to this feature independently
Contributor guide
Research direction
Start by reviewing the reference implementation in #2932 and the core credential-provider API in PR #2976. Then trace the REST catalog, FileIO, and OpenDAL S3/GCS areas named in the issue. Done means refreshable, per-cloud and per-prefix credentials are wired through table FileIO with secure redaction and the stated prefetch, jitter, and backoff behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, gcp, rust
- Domain
- backend-api-design, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100