apache / apache/seatunnel

[Feature][Seatunnel] Introduce support for key vaults such as Azure KeyVault in configs

Open
#10,309 7 comments 0 reactions 0 assignees View on GitHub
feature help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.

### Description

#### Background

Many organizations already rely on centralized secret management systems (key vaults) such as **Azure Key Vault**, **AWS Secrets Manager**, **HashiCorp Vault**, or **GCP Secret Manager** to securely store and rotate credentials. Native support for these systems would significantly improve SeaTunnel’s security posture and enterprise adoption.
#### Proposed Feature

Introduce **native support for external key vaults** in SeaTunnel configuration resolution, starting with **Azure Key Vault**, and designed to be extensible for other providers.

##### Key Capabilities

1. **Secret Reference Syntax in Configs**
Allow configuration values to reference secrets stored in a key vault, for example:

```hocon
password = ${keyvault:azure:my-key-vault/db-password}
```

2. **Pluggable Vault Provider Architecture**

* Define a generic `SecretProvider` interface.
* Implement initial providers:

* Azure Key Vault
* (Future) AWS Secrets Manager
* (Future) HashiCorp Vault
* (Future) GCP Secret Manager

3. **Authentication Support**
For Azure Key Vault:

* Managed Identity
* Service Principal (Client ID / Secret)
* Default Azure Credential chain

4. **Runtime Secret Resolution**

* Secrets resolved at runtime before job execution.
* No secrets persisted in logs, metrics, or checkpoints.

5. **Backward Compatibility**

* Existing configs continue to work unchanged.
* Vault integration is opt-in.

#### Example Use Case

```hocon
env {
execution.parallelism = 2
}

sink {
Jdbc {
url = "jdbc:sqlserver://mydb.database.windows.net:1433"
user = "seatunnel_user"
password = ${keyvault:azure:prod-kv/sql-password}
}
}
```

#### Benefits

* Improved security and compliance (SOC 2, ISO 27001, HIPAA, etc.)
* Centralized secret management and rotation
* Reduced operational risk
* Stronger enterprise and cloud-native adoption of SeaTunnel

#### Additional Considerations

* Caching and TTL for secret lookups
* Failure behavior (e.g., fail-fast if secret is unavailable)
* Clear documentation and examples
* Masking secrets in logs and error messages

#### Open Questions

* Should secret resolution occur at parse time or execution time?
* Should secrets be refreshable for long-running jobs?
* Preferred syntax and naming convention

### Usage Scenario

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named in the issue. Start by locating SeaTunnel’s configuration resolution path and resolving the provider boundary, Azure authentication modes, resolution timing, refresh behavior, and failure handling. Done means an agreed extensible design with syntax, backward compatibility, secret masking, and tests or documentation specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
backend, cloud, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.