[Feature] Support keyless GCS authentication (Application Default Credentials / Workload Identity) for storage vaults
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
Today, an S3-type storage vault with `provider = GCP` can only authenticate with static HMAC keys (`s3.access_key` / `s3.secret_key`). The other credential mechanisms available to vaults (`s3.role_arn`, `s3.credentials_provider_type`) are AWS-specific, so on GCP there is currently no way to run a compute-storage decoupled cluster without minting and distributing long-lived HMAC key pairs.
For deployments on GKE this is a real gap: the platform-native (and increasingly organization-mandated) way to authorize workloads is Workload Identity / Application Default Credentials, where pods obtain short-lived OAuth2 tokens from the metadata server and no static secret exists at all. AWS deployments already enjoy the equivalent via `WEB_IDENTITY`/instance-profile credentials (thanks to the great work in #62788 and #64766) — it would be lovely for GCP users to have parity.
The GCS XML API (which Doris already uses for GCP vaults through the S3-compatible endpoint) accepts `Authorization: Bearer ` as a first-class alternative to HMAC signatures, so this doesn't require a new SDK dependency — "only" a way to resolve ADC tokens and attach them to vault I/O requests across FE/BE/meta-service.
### Use case
`CREATE STORAGE VAULT` on a GKE cluster with Workload Identity, with no static credential anywhere:
```sql
CREATE STORAGE VAULT IF NOT EXISTS gcs_vault
PROPERTIES (
"type" = "S3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "us-central1",
"s3.bucket" = "my-bucket",
"s3.root.path" = "doris",
"provider" = "GCP",
"s3.credentials_provider_type" = "gcp_adc"
);
```
Benefits: no key minting/distribution/rotation, tokens are short-lived, and access control lives entirely in IAM bindings on the bucket — consistent with the keyless direction the vault credential work has been taking on AWS.
### Related issues
#62987 added Google authentication for Iceberg REST catalogs (FE-side); this proposes the analogous capability for storage vaults (data path).
I have a working implementation following the pattern of #64766 (new `CredProviderTypePB` value, shared token provider built on the AWS SDK's HTTP/JSON utilities, bearer header on the S3 client with anonymous credentials) and will open a PR shortly — very happy to adjust the approach based on maintainer feedback.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by reading the related vault credential work in #64766 and the FE-side Google authentication in #62987. Trace CREATE STORAGE VAULT and the credential flow across FE, BE, and meta-service, then verify that GCP ADC or Workload Identity works without static keys and that bearer authentication reaches vault I/O requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, google-cloud, java
- Domain
- backend-api-design, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100