dlt-hub / dlt-hub/dlt

Support more credential sources in dlt CLI

Open
#2,139 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 15h
Merged PRs (30d)
38

Description

### Feature description

Currently, the CLI only works when specifying credentials in 3 out of 6 possible ways:

- [x] secrets.toml
- [x] config.toml
- [x] environment variables
- [ ] [custom config providers](https://dlthub.com/docs/general-usage/credentials/setup/#custom-providers)
- [ ] parameters specified in the pipeline
For example,
```python
pipeline = dlt.pipeline(..., destination=dlt.destinations.clickhouse(credentials=my_credentials))
```
- [ ] dynamically provided credentials
For example, `dlt.secrets["my-secret"] = "my_dynamic_secret_value"`

In practical terms, this makes eg. using a secret manager for storing credentials troublesome, because users need to have one of the 3 static credential sources available locally, which basically defeats the purpose of using an external secret manager.

### Are you a dlt user?

None

### Use case

_No response_

### Proposed solution

Probably creds passed via the `credentials` param should be prioritized, as some of the tutorials use this method of providing credentials (eg. https://dlthub.com/docs/pipelines/sql_database_mssql/load-data-with-python-from-sql_database_mssql-to-mssql#5-next-steps-to-get-your-pipeline-running-in-production), which means if you followed the tutorial, you would end up unable to use `dlt` CLI commands.

### Related issues

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.