dbt-labs / dbt-labs/dbt

[Feature] Apply selection only once when running `dbt build`

Open
#10,593 0 comments 0 reactions 0 assignees View on GitHub
engine:v1 help_wanted performance type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

When running `dbt` commands with `--select` criteria against a project with thousands of models, their resolution can be pretty slow - e.g. one minute for one intersection. But when running `dbt build` the situation is even worse, because the resolution happens three times (once per each underlying command).

I wonder if `dbt build` could trigger `dbt ls` under-the-hood as an initial step, collect the output, and pass the list of found resources to the following commands instead of re-applying `--select`s each time. Potentially it should result in a significant performance improvement.

Sample logs:
```
[2024-08-20, 06:23:28 UTC] {subprocess.py:93} INFO - 06:23:28 Found 14340 models, 31287 data tests, 3 seeds, 2 operations, 7686 sources, 1549 macros
[2024-08-20, 06:36:35 UTC] {subprocess.py:93} INFO - 06:36:35 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:error' does not match any enabled nodes
[2024-08-20, 06:37:33 UTC] {subprocess.py:93} INFO - 06:37:33 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:skipped' does not match any enabled nodes
[2024-08-20, 06:38:31 UTC] {subprocess.py:93} INFO - 06:38:31 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:fail' does not match any enabled nodes
[2024-08-20, 06:51:37 UTC] {subprocess.py:93} INFO - 06:51:37 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:error' does not match any enabled nodes
[2024-08-20, 06:52:36 UTC] {subprocess.py:93} INFO - 06:52:36 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:skipped' does not match any enabled nodes
[2024-08-20, 06:53:35 UTC] {subprocess.py:93} INFO - 06:53:35 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:fail' does not match any enabled nodes
[2024-08-20, 07:06:45 UTC] {subprocess.py:93} INFO - 07:06:45 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:error' does not match any enabled nodes
[2024-08-20, 07:07:44 UTC] {subprocess.py:93} INFO - 07:07:44 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:skipped' does not match any enabled nodes
[2024-08-20, 07:08:43 UTC] {subprocess.py:93} INFO - 07:08:43 The selection criterion '+tag:tag-a,tag:tag-b,config.materialized:view,result:fail' does not match any enabled nodes
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.