apache / apache/doris

[Feature] Support keyless GCS authentication (Application Default Credentials / Workload Identity) for storage vaults

Open
#65,432 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

Today, an S3-type storage vault with `provider = GCP` can only authenticate with static HMAC keys (`s3.access_key` / `s3.secret_key`). The other credential mechanisms available to vaults (`s3.role_arn`, `s3.credentials_provider_type`) are AWS-specific, so on GCP there is currently no way to run a compute-storage decoupled cluster without minting and distributing long-lived HMAC key pairs.

For deployments on GKE this is a real gap: the platform-native (and increasingly organization-mandated) way to authorize workloads is Workload Identity / Application Default Credentials, where pods obtain short-lived OAuth2 tokens from the metadata server and no static secret exists at all. AWS deployments already enjoy the equivalent via `WEB_IDENTITY`/instance-profile credentials (thanks to the great work in #62788 and #64766) — it would be lovely for GCP users to have parity.

The GCS XML API (which Doris already uses for GCP vaults through the S3-compatible endpoint) accepts `Authorization: Bearer ` as a first-class alternative to HMAC signatures, so this doesn't require a new SDK dependency — "only" a way to resolve ADC tokens and attach them to vault I/O requests across FE/BE/meta-service.

### Use case

`CREATE STORAGE VAULT` on a GKE cluster with Workload Identity, with no static credential anywhere:

```sql
CREATE STORAGE VAULT IF NOT EXISTS gcs_vault
PROPERTIES (
"type" = "S3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "us-central1",
"s3.bucket" = "my-bucket",
"s3.root.path" = "doris",
"provider" = "GCP",
"s3.credentials_provider_type" = "gcp_adc"
);
```

Benefits: no key minting/distribution/rotation, tokens are short-lived, and access control lives entirely in IAM bindings on the bucket — consistent with the keyless direction the vault credential work has been taking on AWS.

### Related issues

#62987 added Google authentication for Iceberg REST catalogs (FE-side); this proposes the analogous capability for storage vaults (data path).

I have a working implementation following the pattern of #64766 (new `CredProviderTypePB` value, shared token provider built on the AWS SDK's HTTP/JSON utilities, bearer header on the S3 client with anonymous credentials) and will open a PR shortly — very happy to adjust the approach based on maintainer feedback.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by reading the related vault credential work in #64766 and the FE-side Google authentication in #62987. Trace CREATE STORAGE VAULT and the credential flow across FE, BE, and meta-service, then verify that GCP ADC or Workload Identity works without static keys and that bearer authentication reaches vault I/O requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, google-cloud, java
Domain
backend-api-design, cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.