dlt-hub / dlt-hub/dlt

Add Azure Key Vault support as a VaultDocProvider

Open
#4,076 1 comment 1 reaction 0 assignees View on GitHub
integration
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

dlt currently supports retrieving configuration from secure vaults such as Google Secret Manager and Airflow Variables. The documentation also mentions that contributions for other vault integrations like Azure Key Vault are welcome.

i waned to propose adding Azure Key Vault support as a new VaultDocProvider.

Allow users to load dlt configuration and secrets from Azure Key Vault in the same way they can currently use Google Secret Manager.

This would provide a first-class secret management option for users deploying dlt on Azure.

### Are you a dlt user?

Yes, I'm already a dlt user.

### Use case

Organizations running workloads on Azure often store application secrets in Azure Key Vault and prefer not to duplicate credentials in environment variables or configuration files.

Currently, Azure users do not have a built-in vault integration equivalent to Google Secret Manager.

My case is simple, i'm using fabricspark as backbone of DBT and now for ingestion job i wanted to run DLT job inside fabric notebook and i'm already using AKS which will store source creds such as strip, jdbc, kafka, salesforce etc.

### Proposed solution

Implement an AzureKeyVaultDocProvider based on:
- [azure-keyvault-secrets](https://pypi.org/project/azure-keyvault-secrets/)
- [azure-identity](https://pypi.org/project/azure-identity/)

Authentication could use [DefaultAzureCredential](https://learn.microsoft.com/en-us/dotnet/api/azure.identity.defaultazurecredential?view=azure-dotnet), which supports:

- Local development via az login
- Service principals
- Managed identities
- CI/CD environments

This would follow Azure SDK best practices while keeping authentication configuration minimal.

Needs to check whether azure dependencies should be provided as optional extras.

### Related issues

I couldn't find an existing issue or active PR for Azure Key Vault support, but please let me know if I missed one.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Google Secret Manager VaultDocProvider and its tests, then compare how dlt configuration and secrets are loaded. Review the azure-keyvault-secrets and azure-identity integration and determine how optional dependencies are declared. Done means Azure Key Vault can provide configuration and secrets through the same provider mechanism, with authentication and dependency behavior covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.