apache / apache/seatunnel

[Feature][Connector-V2] Support multiple credential providers for OssFile connector

Open
#10,702 7 comments 0 reactions 1 assignee Claimed by @xxzuo View on GitHub
feature
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.

### Description

Currently, the OssFile connector (both source and sink) only supports a single authentication mode: directly configuring `access_key` and `access_secret` in the job configuration file. This has significant security and usability limitations in cloud environments.

This feature proposes adding support for multiple OSS credential providers through a new configuration option `fs.oss.credentials.provider`, which accepts a list of provider class names and tries them in order (chain pattern) until one succeeds.

**Proposed Authentication Modes:**

| Mode | Provider Class | Description |
|------|---------------|-------------|
| AK/SK from config | (default, no provider needed) | Backward compatible, reads `access_key`/`access_secret` from config file |
| Environment variables | `EnvironmentVariableCredentialsProvider` | Reads `OSS_ACCESS_KEY_ID` / `OSS_ACCESS_KEY_SECRET` from environment variables |
| ECS RAM Role | `OssInstanceProfileCredentialsProvider` | Credential-free access via ECS metadata service, requires `role_name` |
| STS Token | `OssStsCredentialsProvider` | Temporary credentials with `access_key`, `access_secret` and `security_token` |

**New Configuration Options:**

| Option | Type | Required | Description |
|--------|------|----------|-------------|
| `fs.oss.credentials.provider` | `List` | No | Credential provider class names, tried in order |
| `role_name` | `String` | No | ECS RAM role name (for InstanceProfile provider) |
| `security_token` | `String` | No | STS security token (for STS provider) |

**Design Details:**

1. **Backward compatible**: When `fs.oss.credentials.provider` is not configured, the connector falls back to the existing `access_key`/`access_secret` behavior. No breaking changes.

2. **Provider chain**: Multiple providers can be configured as a chain. For example, try environment variables first, then fall back to ECS RAM role:
```hocon
fs.oss.credentials.provider = [
".EnvironmentVariableCredentialsProvider",
"OssInstanceProfileCredentialsProvider"
]
```

3. **hadoop-aliyun integration**:
- Single provider → set directly on `fs.oss.credentials.provider` (native hadoop-aliyun key)
- Multiple providers → use a custom `OssCredentialProviderChain` class that is set as the provider, with the chain list stored in a separate configuration key `fs.oss.credentials.provider.chain`

4. **Wrapper providers**: The OSS SDK's `InstanceProfileCredentialsProvider(String roleName)` and `DefaultCredentialProvider(ak, sk, token)` constructors are incompatible with hadoop-aliyun's reflection mechanism `(URI, Configuration)`. Two wrapper classes are needed:
- `OssInstanceProfileCredentialsProvider(URI, Configuration)` — reads `role_name` from Configuration and delegates to the SDK's InstanceProfileCredentialsProvider
- `OssStsCredentialsProvider(URI, Configuration)` — reads AK/SK/Token from Configuration and delegates to the SDK's DefaultCredentialProvider

5. **Sink factory change**: `access_key` and `access_secret` change from `required` to `optional` in OssFileSinkFactory (Source already has them as optional).

### Usage Scenario

1. **ECS deployment (most common)**: When SeaTunnel runs on Alibaba Cloud ECS instances, users want to avoid hardcoding AK/SK in config files. Using ECS RAM roles allows automatic, credential-free access to OSS with automatic key rotation.

2. **Temporary access**: When third-party applications need temporary OSS access, STS tokens provide time-limited credentials that automatically expire, improving security.

3. **Multi-environment deployment**: Using provider chains allows the same configuration to work across different environments (e.g., ECS in production with RAM role, local development with environment variables).

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.