[BUG] Functional regression: run_query/statement blocks in model body don't share the model's DB connection, breaking the dbt-core dynamic warehouse workaround
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
## Describe the bug
In **dbt-core**, a well-known workaround exists for dynamically switching Snowflake warehouses based on run-time conditions (e.g., full-refresh vs. incremental, or first build vs. subsequent builds). The technique is to call `run_query` or use a `{% call statement() %}` block **inside the model's Jinja body** (outside the `config()` block) to emit a `USE WAREHOUSE ` statement. This works in dbt-core because `run_query` / `statement` blocks share the **same database connection** as the model's own execution.
In **dbt-fusion**, this pattern is broken: the `USE WAREHOUSE` SQL executes on a connection that is **not** the same one used to build the model. By the time Fusion executes the model's SQL, it has moved to a different connection/node, reverting to the default warehouse.
This is a **functional regression** for customers migrating from dbt-core to Fusion who rely on this pattern.
## Example model code
```sql
{{ config(
tags = ['interval_1_hour'],
materialized = 'incremental',
incremental_strategy='append'
) }}
{% if not is_incremental() or flags.FULL_REFRESH %}
{{ use_warehouse(env_var('DBT_WAREHOUSE_LARGE')) }}
{% endif %}
WITH source AS (
SELECT ...
)
SELECT ...
```
Where `use_warehouse` is a macro that calls `run_query` (or uses a `statement` block) to emit `USE WAREHOUSE `.
## Why the workaround exists in dbt-core
The `snowflake_warehouse` config option does not fully cover all dynamic warehouse selection needs:
1. **Full-refresh flag** (`flags.FULL_REFRESH`): Can be handled in the config block, but only if partial parsing is disabled (partial parsing caches the warehouse from the previous parse state, causing issues).
2. **First-time table creation** (i.e., `not is_incremental()` because the table doesn't exist yet): Cannot be handled in the config block because `is_incremental()` evaluates to `false` both on first-run AND on full-refresh — so `snowflake_warehouse` config cannot distinguish between these cases to decide "use the large warehouse for this build."
The `run_query` inline workaround handles **both** cases correctly in dbt-core because the `USE WAREHOUSE` statement and the model SQL run on the same connection.
## Expected behavior
`run_query` / `statement` blocks executed inline within a model's Jinja body should run on the **same database connection** as the model's own SQL, matching dbt-core behavior. This would restore the existing workaround.
Alternatively, Fusion should provide a first-class way to dynamically select a Snowflake warehouse based on both `flags.FULL_REFRESH` **and** whether the model's relation already exists (`is_incremental()` / `load_relation(this)`), so users have a supported migration path away from the workaround.
## Steps to reproduce
1. Create a macro `use_warehouse` that runs `USE WAREHOUSE {{ warehouse }}` via `run_query`.
2. Use it in a model body guarded by `{% if not is_incremental() %}`.
3. Run `dbt run` on a relation that does not yet exist.
4. Observe that the model builds using the **default** warehouse, not the larger one specified by `use_warehouse`.
5. Compare to dbt-core behavior where the larger warehouse is correctly used.
## Is this a discrepancy between dbt Fusion Engine and dbt Core?
- [x] YES
- [ ] NO
## Impact
Customers with large incremental models that need a bigger warehouse for full-refresh or first-build runs are silently using the wrong (smaller) warehouse in Fusion, leading to slower runtimes or query failures. This was surfaced by customer **Norlys** during a Fusion migration.
## Version
Observed in current Fusion preview versions (confirmed via customer debug logs showing the `USE WAREHOUSE` executes but the model build reverts to default warehouse).
Contributor guide
Assessment
This issue has not been assessed yet.