Support URI-aware operator injection for object storage
- Vorherrschende Sprache
- Rust
- Sterne
- 197
- Forks
- 92
- Ø Merge
- 1 T. 20 Std.
- Gemergte PRs (30 T.)
- 108
Beschreibung
### Motivation
`FileIO` currently accepts object-store credentials as static properties. A long-lived `FileIO` cannot observe rotated temporary credentials without being rebuilt.
For example, an external process may atomically update a JSON file containing AK/SK/session token and expiration. The same `FileIO` should reload it when the cached credential expires. This is needed by the [Milvus external-table integration](https://github.com/milvus-io/milvus/issues/50632), where explicitly configured delegated authentication must also [fail closed](https://github.com/milvus-io/milvus-storage/issues/617).
### Proposal
Following iceberg-rust's [`CustomAwsCredentialLoader`](https://github.com/apache/iceberg-rust/blob/main/crates/storage/opendal/src/s3.rs), allow object-store builders used by `FileIO` to accept provider-specific reqsign credential providers.
- The provider returns the native credential and its expiration where supported. reqsign caches it and invokes the provider again when it becomes invalid.
- An explicitly configured provider replaces the default credential chain; refresh errors are returned without falling back to another identity.
- Existing property-based configuration remains unchanged. STS, OIDC, ARN, impersonation, and credential-broker implementations stay outside paimon-rust.
- Support S3 first, then use the same pattern for OSS, GCS, and Azure where OpenDAL / reqsign expose equivalent hooks.
Reloading credentials revoked before their declared expiration requires structured invalidation and retry support from OpenDAL / reqsign. Paimon should not implement this by matching cloud error strings.
Beitragsleitfaden
Rechercherichtung
Beginne mit den Object-Store-Buildern von FileIO und der vorhandenen eigenschaftsbasierten Credential-Konfiguration und untersuche anschließend die im Vorschlag beschriebenen Provider-Hooks von OpenDAL/reqsign. Als abgeschlossen gilt die Aufgabe, wenn S3 einen expliziten erneuerbaren Provider unterstützt, dessen Ablaufzeit verwendet und Refresh-Fehler zurückgibt, ohne auf einen Fallback zurückzugreifen; füge Tests für die Provider-Erneuerung und das Fail-Closed-Verhalten hinzu.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- rust
- Bereich
- authentication, cloud
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 45/100