[Feature][Seatunnel] Introduce support for key vaults such as Azure KeyVault in configs
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
### Search before asking
- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
#### Background
Many organizations already rely on centralized secret management systems (key vaults) such as **Azure Key Vault**, **AWS Secrets Manager**, **HashiCorp Vault**, or **GCP Secret Manager** to securely store and rotate credentials. Native support for these systems would significantly improve SeaTunnel’s security posture and enterprise adoption.
#### Proposed Feature
Introduce **native support for external key vaults** in SeaTunnel configuration resolution, starting with **Azure Key Vault**, and designed to be extensible for other providers.
##### Key Capabilities
1. **Secret Reference Syntax in Configs**
Allow configuration values to reference secrets stored in a key vault, for example:
```hocon
password = ${keyvault:azure:my-key-vault/db-password}
```
2. **Pluggable Vault Provider Architecture**
* Define a generic `SecretProvider` interface.
* Implement initial providers:
* Azure Key Vault
* (Future) AWS Secrets Manager
* (Future) HashiCorp Vault
* (Future) GCP Secret Manager
3. **Authentication Support**
For Azure Key Vault:
* Managed Identity
* Service Principal (Client ID / Secret)
* Default Azure Credential chain
4. **Runtime Secret Resolution**
* Secrets resolved at runtime before job execution.
* No secrets persisted in logs, metrics, or checkpoints.
5. **Backward Compatibility**
* Existing configs continue to work unchanged.
* Vault integration is opt-in.
#### Example Use Case
```hocon
env {
execution.parallelism = 2
}
sink {
Jdbc {
url = "jdbc:sqlserver://mydb.database.windows.net:1433"
user = "seatunnel_user"
password = ${keyvault:azure:prod-kv/sql-password}
}
}
```
#### Benefits
* Improved security and compliance (SOC 2, ISO 27001, HIPAA, etc.)
* Centralized secret management and rotation
* Reduced operational risk
* Stronger enterprise and cloud-native adoption of SeaTunnel
#### Additional Considerations
* Caching and TTL for secret lookups
* Failure behavior (e.g., fail-fast if secret is unavailable)
* Clear documentation and examples
* Masking secrets in logs and error messages
#### Open Questions
* Should secret resolution occur at parse time or execution time?
* Should secrets be refreshable for long-running jobs?
* Preferred syntax and naming convention
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named in the issue. Start by locating SeaTunnel’s configuration resolution path and resolving the provider boundary, Azure authentication modes, resolution timing, refresh behavior, and failure handling. Done means an agreed extensible design with syntax, backward compatibility, secret masking, and tests or documentation specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java
- Domain
- backend, cloud, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100