dbt-labs / dbt-labs/dbt

[Feature] Add ability to import/include YAML from other files

Open
#9,695 11 comments 48 reactions 0 assignees View on GitHub
engine:v1 paper_cut Refinement type:feature yaml
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

I wonder if dbt would benefit from providing the ability to include YAML snippets from other files.

Home Assistant (a popular python tool, leveraging YAML for config), [provides a few additional YAML constructors](https://www.home-assistant.io/docs/configuration/splitting_configuration/) ([quick overview of constructors](https://matthewpburruss.com/post/yaml/#defining-pyyaml-constructors-going-from-yaml-to-python)).

The most interesting one would be `!include` which allows to define values of keys in YAML from other YAML files
The implementation of Home Assistant is [in this source file](https://github.com/home-assistant/core/blob/b629f5c143b4d2054b77913c550e10098825ed8b/homeassistant/util/yaml/loader.py#L507).

While we can define some logic in Jinja in some YAML files, we are limited today to defining a single entry at a time (e.g. a string or a bool). With this approach we can also store config for nested fields.

Based on my first observations, this technique wouldn't mess with the current dbt code as the YAML is rendered before we start any "real" dbt parsing.

### Describe alternatives you've considered

Copy pasting the same logic in YAML as we do today

### Who will this benefit?

- people who want to DRY their YAML files (e.g common tests for columns, common tags etc...)
- people who want to define the config of `dbt_project.yml` from different files, with different code owners in git
- people following [this issue](https://github.com/dbt-labs/dbt-core/issues/2955)

### Are you interested in contributing this feature?

Yes

### Anything else?

I got an early prototype working locally and can share some of the code if we want to implement this feature.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.