apache / apache/doris

[Feature] Add Alibaba Cloud OSS C++ SDK stack

Open
#65,793 2 comments 0 reactions 0 assignees View on GitHub
kind/feature
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 21h
Merged PRs (30d)
531

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

Add native Alibaba Cloud OSS support as a first-class storage backend in Doris, using the official Alibaba Cloud OSS C++ SDK instead of the existing S3-compatible (AWS SDK) path.

### Use case

Doris users running on Alibaba Cloud ECS face three problems with the current S3-compatible OSS path:

1. **No credential-free access** — The AWS SDK cannot talk to the ECS instance metadata service (100.100.100.200). Users must manage static AK/SK keys even when an ECS RAM role is attached to the instance.

2. **No STS AssumeRole** — Cross-account access and least-privilege patterns require `acs:ram::account:role/name` ARNs which are Alibaba-specific and not reachable via the AWS STS endpoint.

3. **No RRSA (OIDC)** — Kubernetes workloads on ACK use RRSA (RAM Roles for Service Accounts) for pod-level identity. The AWS SDK has no awareness of Alibaba's OIDC token file mechanism.

**Target users:**

- Doris cloud-mode deployments on Alibaba Cloud ECS
- Enterprises using RAM roles for zero-credential rotation policies
- ACK (Alibaba Container Service for Kubernetes) workloads using RRSA
- Cross-account OSS access patterns using STS AssumeRole

**Concrete scenarios:**

- Storage vaults backed by OSS with ECS instance profile (no AK/SK in config)
- Cloud recycler cleaning up OSS garbage with RAM role credentials
- Catalog/external table reads from OSS with STS-assumed least-privilege role

### Related issues

No existing issue. This feature was discussed informally and a draft PR (#61329) has been submitted. This issue formalizes the feature request and tracks the work across 4 incremental PRs:

- PR-1: #61329 FE `fe-filesystem-oss` SPI — ECS RAM role, AssumeRole, RRSA, `oss.credentials_provider` type-based dispatch

- PR-2: #65794 Thirdparty C++ SDK stack — apr, tea-cpp, credentials-cpp, openapi-v2, sts-20150401, alibabacloud-oss-cpp-sdk

- PR-3: BE native OSS filesystem — `OSSFileSystem`, `OSSFileReader`, `OSSFileWriter`, credential providers, `enable_oss_native_sdk` config flag
- PR-4: Cloud recycler OSS accessor — `OSSAccessor`, `OssObjClient`, reuses shared `delete_objects_recursively_()` helper

**Design principle:** All changes are additive and gated behind `enable_oss_native_sdk = false` (default). Existing S3-compatible OSS path is fully preserved when the flag is off.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with draft PR #61329 and the third-party SDK work in PR #65794, then trace the named fe-filesystem-oss SPI and the planned OSSFileSystem, OSSFileReader, OSSFileWriter, credential providers, and OSSAccessor components. Done means the four incremental PRs provide native Alibaba OSS support behind enable_oss_native_sdk=false while preserving the existing S3-compatible path.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.