dbt-labs / dbt-labs/dbt

[Feature] Namespaced variables: allow platform-owned vars and project-owned vars to coexist

Open
#15,478 1 comment 0 reactions 0 assignees View on GitHub
triage
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

# Namespaced variables: allow platform-owned vars and project-owned vars to coexist

## Context

This picks up the "namespaced variables" thread that was deliberately descoped from #2955. In [that issue][2955-comment], @graciegoheen split the discussion into three problems and only carried problems 1 & 2 forward into an implementation issue (#11144, shipped via #12488, root-level `vars.yml` parsed before `dbt_project.yml`):

1. Define project-wide vars outside `dbt_project.yml`
2. Reference vars *in* `dbt_project.yml` at parse-time
3. **Namespaced variables** which was left open, with an invitation to open a new issue

This is that issue.

## Use case

A customer runs a platform team that centrally manages a set of variables across many downstream dbt projects (e.g. environment config, shared business logic thresholds, connection/behavior flags). Individual project teams need to:

- Define and use their **own** variables freely, in their own `dbt_project.yml` or `vars.yml` — since #12488, projects can already use both files for their own vars.
- **Read** the platform-managed variables.
- Be **prevented from overriding or redefining** the platform-managed variables - today, since `vars` resolve as one flat namespace, any project team can accidentally (or deliberately) shadow a platform var by declaring a var with the same name, and there's no mechanism to detect or block that collision.

Their preferred shape is to keep `vars` in **both** `dbt_project.yml` (project-owned) and `vars.yml` (platform-owned) simultaneously, not choose one or the other, with dbt enforcing a boundary between the two: platform vars win / are locked, and a name collision from the project side raises an error rather than silently overriding.

When this came up, @graciegoheen pointed out that this is really the namespacing problem (`#3` above), not the "reference vars in `dbt_project.yml`" problem (`#2`) that #11144 solved - a governance/ownership boundary between two variable scopes, not just a parse-order fix.

## Prior art from #2955

A few concrete proposals came out of the multi-year discussion on #2955 that are relevant here:

- **Dict-based namespacing (workaround, no core changes)** - nest vars under a namespace key and access via a custom macro that falls back to a shared `defaults` namespace:
```yaml
vars:
defaults:
key1: value_a
platform:
key1: value1
team_a:
key1: value_x
```
This works today but has no enforcement - any project can still edit the `platform` block.

- **Multiple var files, one per namespace** e.g. a `vars/` directory (`vars/platform.yml`, `vars/team_a.yml`), with dbt erroring if the same top-level key/namespace appears in more than one file. This is the closest prior proposal to an ownership boundary, but was never fleshed out for cross-file precedence or locking.

- **Folder-level / mini-project configs** - teams wanting to own their own config without touching project-level files, for governance and merge-conflict reasons rather than strict variable scoping. Related motivation, distinct mechanism.

- **Sub-projects/packages as the existing namespacing mechanism** - the standing workaround (and the one @jtcohen6 suggested back in #2955) is to split a large project into packages, since packages already get their own var namespace. This doesn't help the platform-vs-project ownership case, since the platform team doesn't want to own a package per downstream team.

## Historical note

Hierarchical/namespaced vars scoped by package existed in the `dbt_project.yml` v2 config spec pre-0.17 and were removed. @drewbanin's writeup in [#2300][2300-comment] is the best record of why: variable resolution became hard to reason about once vars could be scoped by folder path in addition to package, particularly for files like `schema.yml` that don't obviously inherit a folder's scope. Any new namespacing design should explicitly define resolution order and avoid reintroducing that ambiguity — a two-tier (platform vs. project) namespace with an explicit ownership/override rule is a much narrower scope than the old folder-based hierarchy, which may sidestep most of that problem.

## Open questions

- Does "platform-owned and locked" need to be a first-class concept (e.g. a `vars.yml` block or file that project-level config cannot shadow), or is a collision **error** (rather than silent override) sufficient?
- Should this build on the `vars.yml` mechanism from #11144/#12488 (e.g. platform vars live in `vars.yml`, project vars stay in `dbt_project.yml`, and dbt errors on any name collision between the two), or does it need a distinct namespace syntax?
- How does this interact with packages, which already have their own implicit namespace?
- What's the right behavior in dbt Cloud/CI contexts where a platform team manages `vars.yml` centrally but project repos are otherwise owned by individual teams?

## References

- #2955 (original vars.yml / namespacing discussion)
- [#2955 comment splitting into three problems][2955-comment]
- #2300, [comment on original removal of hierarchical vars][2300-comment]
- #11144 (implementation issue for problems 1 & 2)
- #12488 (vars.yml implementation PR)

[2955-comment]: https://github.com/dbt-labs/dbt-core/issues/2955#issuecomment-2539750686
[2300-comment]: https://github.com/dbt-labs/dbt-core/issues/2300#issuecomment-610610316

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.