[Bug] Show objects in schema queries run for all models running in parallel before the schema cache is populated
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this a new bug in dbt-core?
- [x] I believe this is a new bug in dbt-core
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Current Behavior
At the start of a dbt run, dbt fetches all the objects in a schema. In the debug logs, you can see the queries are run 6 times, once for each of the staging models at the start of the jaffle_shop DAG.
This is made worse for me in particular because I have a _lot_ of schemas due to a benchmarking project
```
Query executed on node model.jaffle_shop_7.stg_customers:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000
Query executed on node model.jaffle_shop_7.stg_products:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000
Query executed on node model.jaffle_shop_7.stg_orders:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000
Query executed on node model.jaffle_shop_7.stg_supplies:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000
Query executed on node model.jaffle_shop_7.stg_locations:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000
Query executed on node model.jaffle_shop_7.stg_order_items:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000
```
My schema contains more than 10k objects so it's even worse for me:
```
Query executed on node model.jaffle_shop_7.stg_supplies:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'INT__BASE_TEST_1024'
Query executed on node model.jaffle_shop_7.stg_orders:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'INT__BASE_TEST_1024'
Query executed on node model.jaffle_shop_7.stg_locations:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'INT__BASE_TEST_1024'
Query executed on node model.jaffle_shop_7.stg_products:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'INT__BASE_TEST_1024'
Query executed on node model.jaffle_shop_7.stg_order_items:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'INT__BASE_TEST_1024'
Query executed on node model.jaffle_shop_7.stg_customers:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'INT__BASE_TEST_1024'
Query executed on node model.jaffle_shop_7.stg_customers:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'MART__BAND_GAGE_31'
Query executed on node model.jaffle_shop_7.stg_products:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'MART__BAND_GAGE_31'
Query executed on node model.jaffle_shop_7.stg_locations:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'MART__BAND_GAGE_31'
Query executed on node model.jaffle_shop_7.stg_order_items:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'MART__BAND_GAGE_31'
Query executed on node model.jaffle_shop_7.stg_supplies:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'MART__BAND_GAGE_31'
Query executed on node model.jaffle_shop_7.stg_products:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'MART__DOPE_FARE_894'
Query executed on node model.jaffle_shop_7.stg_customers:
SHOW OBJECTS IN SCHEMA "ANALYTICS"."DBT_JLABES" LIMIT 10000 FROM 'MART__DOPE_FARE_894'
```
These queries should only be run once for the whole project, regardless of the number of nodes which are able to be run in parallel at the start of a build.
Running `dbt run --threads 1 --log-level debug` reduces the number of times the queries are run from 6 to 2 (ideally it should only be 1 to fully respect the --threads flag, but that's a different problem), and makes the other queries queue and then have a very short execution time:
With 1 thread: note that most stg_ models complete in <1 sec
```
(.dbtcorefs) joel@Joel-Labes jaffle_shop_7 % dbt run --threads 1
dbt-core 2.0.0-alpha.1
Loading ~/.dbt/profiles.yml
Loading packages.yml
Succeeded [ 18.93s] model dbt_jlabes.stg_customers (view)
Succeeded [ 0.91s] model dbt_jlabes.stg_order_items (view)
Succeeded [ 0.52s] model dbt_jlabes.stg_products (view)
Succeeded [ 0.54s] model dbt_jlabes.stg_orders (view)
Succeeded [ 0.57s] model dbt_jlabes.stg_supplies (view)
Succeeded [ 1.16s] model dbt_jlabes.products (table)
Succeeded [ 1.53s] model dbt_jlabes.supplies (table)
Succeeded [ 2.42s] model dbt_jlabes.order_items (table)
Succeeded [ 1.27s] model dbt_jlabes.orders (table)
Succeeded [ 2.25s] model dbt_jlabes.customers (table)
Succeeded [ 30.18s] model dbt_jlabes.stg_locations (view)
Succeeded [ 1.20s] model dbt_jlabes.locations (table)
========================================================================== Execution Summary ===========================================================================
Finished 'run' successfully for target 'snowflake' [34.4s]
Processed: 12 models
Summary: 12 total | 12 success
```
Without constrained thread count - note that all stg_ models have >20sec runtime because they all run the slow metadata queries
```
(.dbtcorefs) joel@Joel-Labes jaffle_shop_7 % dbt run
dbt-core 2.0.0-alpha.1
Loading ~/.dbt/profiles.yml
Loading packages.yml
Succeeded [ 19.97s] model dbt_jlabes.stg_orders (view)
Succeeded [ 21.84s] model dbt_jlabes.stg_order_items (view)
Succeeded [ 21.92s] model dbt_jlabes.stg_locations (view)
Succeeded [ 1.20s] model dbt_jlabes.locations (table)
Succeeded [ 23.66s] model dbt_jlabes.stg_supplies (view)
Succeeded [ 24.82s] model dbt_jlabes.stg_customers (view)
Succeeded [ 1.26s] model dbt_jlabes.supplies (table)
Succeeded [ 25.51s] model dbt_jlabes.stg_products (view)
Succeeded [ 1.09s] model dbt_jlabes.products (table)
Succeeded [ 1.26s] model dbt_jlabes.order_items (table)
Succeeded [ 1.42s] model dbt_jlabes.orders (table)
Succeeded [ 1.64s] model dbt_jlabes.customers (table)
========================================================================== Execution Summary ===========================================================================
Finished 'run' successfully for target 'snowflake' [33.9s]
Processed: 12 models
Summary: 12 total | 12 success
```
This doesn't negatively impact dbt runtime, but it does lead to excessive queries being run against the remote warehouse unnecessarily.
### Expected Behavior
Schema cache population queries only run once; other models wait for it to finish and share its results instead of firing off their own redundant copies of the query
### Steps To Reproduce
Run `dbt init`
Run `dbt run --log-level debug`
Note multiple SHOW OBJECTS IN SCHEMA queries, one per staging node
Run `dbt run --threads 1 --log-level debug`
Note relatively fewer SHOW OBJECTS IN SCHEMA queries, and that the other staging nodes wait to start executing until the results come back
### Relevant log output
```shell
```
### Environment
```markdown
- OS: MacOS
- dbt: dbt-core 2.0.0-alpha.1
```
### Which database adapter are you using with dbt?
snowflake
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.