Profile config files silently ignored without .dlt/.workspace marker
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 605
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
Problem
When profile-scoped config files like dev.config.toml or prod.config.toml exist in .dlt/, they are silently ignored unless a .dlt/.workspace marker file is present. There is no warning or error indicating that these files are not being loaded.
This leads to confusing behavior where:
- You split destination config into
dev.config.tomlandprod.config.toml(e.g., differentbucket_urlper environment) - Remove the shared value from
config.tomlto avoid conflicts - Pipeline fails with
ConfigFieldMissingExceptionforbucket_url - The error message lists all providers searched but gives no hint that profile config files exist on disk but are not being loaded
Root cause
Without .dlt/.workspace, the run context falls back to plain RunContext instead of WorkspaceRunContext. The profile-scoped TOML providers (ProfileSecretsTomlProvider, etc.) are never registered, so dev.config.toml / prod.config.toml are never read -- even though they're sitting right there in .dlt/.
Expected behavior
At minimum, dlt should warn when .dlt/{profile}.config.toml or .dlt/{profile}.secrets.toml files exist but profile loading is inactive because .dlt/.workspace is missing. Something like:
WARNING: Found profile config files (dev.config.toml, prod.config.toml) in .dlt/
but workspace is not initialized. These files will not be loaded.
Create .dlt/.workspace to enable profile support.
Ideally, the ConfigFieldMissingException output should mention the unloaded profile files when they exist on disk and could contain the missing value.
Steps to reproduce
# Create profile configs
cat > .dlt/dev.config.toml << 'TOML'
[destination.filesystem]
bucket_url = "./out/local-data"
TOML
cat > .dlt/prod.config.toml << 'TOML'
[destination.filesystem]
bucket_url = "s3://my-bucket"
TOML
# Remove bucket_url from shared config (since it's now per-profile)
# .dlt/config.toml has no [destination.filesystem] section
# Do NOT create .dlt/.workspace
# Run pipeline -- fails with ConfigFieldMissingException for bucket_url
uv run python my_pipeline.py
Fix: touch .dlt/.workspace -- then profile configs load correctly.
Related
- #3788 --
dlt profilecommand should always be available with guidance when workspace is not initialized (discoverability side of the same problem)
Environment
- dlt version: 1.24.0
- dlthub version: 0.22.1
- Python: 3.13
- OS: macOS
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace how RunContext and WorkspaceRunContext register ProfileSecretsTomlProvider and related profile-scoped providers when .dlt/.workspace is absent. Reproduce the missing bucket_url case, then determine where a warning or ConfigFieldMissingException hint can be added; done means existing profile files are surfaced without changing behavior when workspace support is enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, developer-experience
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100