dlt-hub / dlt-hub/dlt

Profile config files silently ignored without .dlt/.workspace marker

Open
#3,789 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

Problem

When profile-scoped config files like dev.config.toml or prod.config.toml exist in .dlt/, they are silently ignored unless a .dlt/.workspace marker file is present. There is no warning or error indicating that these files are not being loaded.

This leads to confusing behavior where:

  1. You split destination config into dev.config.toml and prod.config.toml (e.g., different bucket_url per environment)
  2. Remove the shared value from config.toml to avoid conflicts
  3. Pipeline fails with ConfigFieldMissingException for bucket_url
  4. The error message lists all providers searched but gives no hint that profile config files exist on disk but are not being loaded
Root cause

Without .dlt/.workspace, the run context falls back to plain RunContext instead of WorkspaceRunContext. The profile-scoped TOML providers (ProfileSecretsTomlProvider, etc.) are never registered, so dev.config.toml / prod.config.toml are never read -- even though they're sitting right there in .dlt/.

Expected behavior

At minimum, dlt should warn when .dlt/{profile}.config.toml or .dlt/{profile}.secrets.toml files exist but profile loading is inactive because .dlt/.workspace is missing. Something like:

WARNING: Found profile config files (dev.config.toml, prod.config.toml) in .dlt/ 
but workspace is not initialized. These files will not be loaded. 
Create .dlt/.workspace to enable profile support.

Ideally, the ConfigFieldMissingException output should mention the unloaded profile files when they exist on disk and could contain the missing value.

Steps to reproduce

# Create profile configs
cat > .dlt/dev.config.toml << 'TOML'
[destination.filesystem]
bucket_url = "./out/local-data"
TOML

cat > .dlt/prod.config.toml << 'TOML'
[destination.filesystem]
bucket_url = "s3://my-bucket"
TOML

# Remove bucket_url from shared config (since it's now per-profile)
# .dlt/config.toml has no [destination.filesystem] section

# Do NOT create .dlt/.workspace

# Run pipeline -- fails with ConfigFieldMissingException for bucket_url
uv run python my_pipeline.py

Fix: touch .dlt/.workspace -- then profile configs load correctly.

Related

  • #3788 -- dlt profile command should always be available with guidance when workspace is not initialized (discoverability side of the same problem)

Environment

  • dlt version: 1.24.0
  • dlthub version: 0.22.1
  • Python: 3.13
  • OS: macOS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace how RunContext and WorkspaceRunContext register ProfileSecretsTomlProvider and related profile-scoped providers when .dlt/.workspace is absent. Reproduce the missing bucket_url case, then determine where a warning or ConfigFieldMissingException hint can be added; done means existing profile files are surfaced without changing behavior when workspace support is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, developer-experience
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.