Google Secrets provider ignores only_toml_fragments=false when initialized during source resolution
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
dlt 1.27.0
### Describe the problem
When Google Secret Manager is enabled as a config provider, source argument resolution can initialize the live `GoogleSecretsProvider` with default vault settings instead of the resolved values from `.dlt/config.toml`.
In particular, this configuration:
```toml
[providers]
enable_google_secrets = true
[providers.google_secrets]
only_secrets = true
only_toml_fragments = false
list_secrets = false
```
> same behavior when we set `only_secrets = false`
should make the provider probe scalar GSM secrets. For a source argument like:
```python
@dlt.source(name="my_source")
def my_source(api_key: str = dlt.secrets.value):
...
```
I expected dlt to make a backend request for a scalar secret such as:
```text
sources-my_source-api_key
```
Instead, when the source invocation is the path that first creates/initializes the provider, DEBUG logs only show fragment-style requests, for example:
```text
dlt_secrets_toml
sources
sources-my_source
sources-runtime
sources-source
sources-api_key
```
The final missing-config error still lists `(Google Secrets, sources-my_source-api_key)` among the tried keys, but the vault debug logs never show an actual backend request for that scalar key.
The live provider confirms the issue: `GoogleSecretsProvider.only_toml_fragments` is `True`, despite `.dlt/config.toml` setting it to `false`.
### Expected behavior
`GoogleSecretsProvider` should be constructed with the resolved `[providers.google_secrets]` settings from `.dlt/config.toml`, regardless of whether provider initialization is triggered during a source/resource config resolution path.
With `only_toml_fragments = false`, dlt should probe scalar secrets such as:
```text
sources-my_source-api_key
```
### Steps to reproduce
1. Configure Google Secrets in `.dlt/config.toml`:
```toml
[providers]
enable_google_secrets = true
[providers.google_secrets]
only_secrets = true
only_toml_fragments = false
list_secrets = false
```
2. Configure Google credentials via `.dlt/secrets.toml`, environment variables, or application default credentials.
3. Define and invoke a source directly:
```python
import dlt
@dlt.source(name="my_source")
def my_source(api_key: str = dlt.secrets.value):
return []
my_source()
```
4. Inspect the live provider:
```python
from dlt.common.configuration.container import Container
from dlt.common.configuration.specs import PluggableRunContext
for provider in Container()[PluggableRunContext].providers.providers:
if provider.name == "Google Secrets":
print(provider.only_secrets, provider.only_toml_fragments, provider.list_secrets)
```
Observed:
```text
True True False
```
Expected:
```text
True False False
```
5. As a workaround, pre-resolve provider configuration before invoking the source:
```python
from dlt.common.configuration.resolve import resolve_configuration
from dlt.common.configuration.specs.config_providers_context import ConfigProvidersConfiguration
resolve_configuration(ConfigProvidersConfiguration())
```
After this, the live provider is constructed with `only_toml_fragments = False`, and scalar key requests are made.
### Operating system
macOS
### Runtime environment
Local
### Python version
3.13
### dlt data source
Custom `@dlt.source` using a source argument marked with `dlt.secrets.value`.
### dlt destination
Not destination-specific. The issue happens during source configuration resolution before loading.
### Other deployment details
Google Secret Manager provider enabled via local `.dlt/config.toml`.
### Additional information
This appears to be caused by provider bootstrap resolving `ConfigProvidersConfiguration` while a source `ConfigSectionContext` is active.
`enable_google_secrets` is a top-level field and resolves correctly, but nested settings like `providers.google_secrets.only_toml_fragments` can be looked up with source-prefixed paths and fall back to defaults. The result is that the live `GoogleSecretsProvider` is constructed with `VaultProviderConfiguration` defaults.
A focused fix is to resolve extra provider bootstrap configuration under an empty section context, for example by wrapping `_extra_providers()` resolution in `inject_section(ConfigSectionContext(), merge_existing=False)`.
Contributor guide
Assessment
This issue has not been assessed yet.