dlt-hub / dlt-hub/dlt

Google Secrets provider ignores only_toml_fragments=false when initialized during source resolution

Open
#3,969 0 comments 0 reactions 1 assignee Claimed by @rudolfix View on GitHub
bug needs decision
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

dlt 1.27.0

### Describe the problem

When Google Secret Manager is enabled as a config provider, source argument resolution can initialize the live `GoogleSecretsProvider` with default vault settings instead of the resolved values from `.dlt/config.toml`.

In particular, this configuration:

```toml
[providers]
enable_google_secrets = true

[providers.google_secrets]
only_secrets = true
only_toml_fragments = false
list_secrets = false
```
> same behavior when we set `only_secrets = false`

should make the provider probe scalar GSM secrets. For a source argument like:

```python
@dlt.source(name="my_source")
def my_source(api_key: str = dlt.secrets.value):
...
```

I expected dlt to make a backend request for a scalar secret such as:

```text
sources-my_source-api_key
```

Instead, when the source invocation is the path that first creates/initializes the provider, DEBUG logs only show fragment-style requests, for example:

```text
dlt_secrets_toml
sources
sources-my_source
sources-runtime
sources-source
sources-api_key
```

The final missing-config error still lists `(Google Secrets, sources-my_source-api_key)` among the tried keys, but the vault debug logs never show an actual backend request for that scalar key.

The live provider confirms the issue: `GoogleSecretsProvider.only_toml_fragments` is `True`, despite `.dlt/config.toml` setting it to `false`.

### Expected behavior

`GoogleSecretsProvider` should be constructed with the resolved `[providers.google_secrets]` settings from `.dlt/config.toml`, regardless of whether provider initialization is triggered during a source/resource config resolution path.

With `only_toml_fragments = false`, dlt should probe scalar secrets such as:

```text
sources-my_source-api_key
```

### Steps to reproduce

1. Configure Google Secrets in `.dlt/config.toml`:

```toml
[providers]
enable_google_secrets = true

[providers.google_secrets]
only_secrets = true
only_toml_fragments = false
list_secrets = false
```

2. Configure Google credentials via `.dlt/secrets.toml`, environment variables, or application default credentials.

3. Define and invoke a source directly:

```python
import dlt

@dlt.source(name="my_source")
def my_source(api_key: str = dlt.secrets.value):
return []

my_source()
```

4. Inspect the live provider:

```python
from dlt.common.configuration.container import Container
from dlt.common.configuration.specs import PluggableRunContext

for provider in Container()[PluggableRunContext].providers.providers:
if provider.name == "Google Secrets":
print(provider.only_secrets, provider.only_toml_fragments, provider.list_secrets)
```

Observed:

```text
True True False
```

Expected:

```text
True False False
```

5. As a workaround, pre-resolve provider configuration before invoking the source:

```python
from dlt.common.configuration.resolve import resolve_configuration
from dlt.common.configuration.specs.config_providers_context import ConfigProvidersConfiguration

resolve_configuration(ConfigProvidersConfiguration())
```

After this, the live provider is constructed with `only_toml_fragments = False`, and scalar key requests are made.

### Operating system

macOS

### Runtime environment

Local

### Python version

3.13

### dlt data source

Custom `@dlt.source` using a source argument marked with `dlt.secrets.value`.

### dlt destination

Not destination-specific. The issue happens during source configuration resolution before loading.

### Other deployment details

Google Secret Manager provider enabled via local `.dlt/config.toml`.

### Additional information

This appears to be caused by provider bootstrap resolving `ConfigProvidersConfiguration` while a source `ConfigSectionContext` is active.

`enable_google_secrets` is a top-level field and resolves correctly, but nested settings like `providers.google_secrets.only_toml_fragments` can be looked up with source-prefixed paths and fall back to defaults. The result is that the live `GoogleSecretsProvider` is constructed with `VaultProviderConfiguration` defaults.

A focused fix is to resolve extra provider bootstrap configuration under an empty section context, for example by wrapping `_extra_providers()` resolution in `inject_section(ConfigSectionContext(), merge_existing=False)`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.