dbt-labs / dbt-labs/dbt

[BUG] Functional regression: run_query/statement blocks in model body don't share the model's DB connection, breaking the dbt-core dynamic warehouse workaround

Open
#14,566 5 comments 0 reactions 1 assignee Assigned to @akbog View on GitHub
area:engine engine:v2 type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

## Describe the bug

In **dbt-core**, a well-known workaround exists for dynamically switching Snowflake warehouses based on run-time conditions (e.g., full-refresh vs. incremental, or first build vs. subsequent builds). The technique is to call `run_query` or use a `{% call statement() %}` block **inside the model's Jinja body** (outside the `config()` block) to emit a `USE WAREHOUSE ` statement. This works in dbt-core because `run_query` / `statement` blocks share the **same database connection** as the model's own execution.

In **dbt-fusion**, this pattern is broken: the `USE WAREHOUSE` SQL executes on a connection that is **not** the same one used to build the model. By the time Fusion executes the model's SQL, it has moved to a different connection/node, reverting to the default warehouse.

This is a **functional regression** for customers migrating from dbt-core to Fusion who rely on this pattern.

## Example model code

```sql
{{ config(
tags = ['interval_1_hour'],
materialized = 'incremental',
incremental_strategy='append'
) }}

{% if not is_incremental() or flags.FULL_REFRESH %}
{{ use_warehouse(env_var('DBT_WAREHOUSE_LARGE')) }}
{% endif %}

WITH source AS (
SELECT ...
)
SELECT ...
```

Where `use_warehouse` is a macro that calls `run_query` (or uses a `statement` block) to emit `USE WAREHOUSE `.

## Why the workaround exists in dbt-core

The `snowflake_warehouse` config option does not fully cover all dynamic warehouse selection needs:

1. **Full-refresh flag** (`flags.FULL_REFRESH`): Can be handled in the config block, but only if partial parsing is disabled (partial parsing caches the warehouse from the previous parse state, causing issues).
2. **First-time table creation** (i.e., `not is_incremental()` because the table doesn't exist yet): Cannot be handled in the config block because `is_incremental()` evaluates to `false` both on first-run AND on full-refresh — so `snowflake_warehouse` config cannot distinguish between these cases to decide "use the large warehouse for this build."

The `run_query` inline workaround handles **both** cases correctly in dbt-core because the `USE WAREHOUSE` statement and the model SQL run on the same connection.

## Expected behavior

`run_query` / `statement` blocks executed inline within a model's Jinja body should run on the **same database connection** as the model's own SQL, matching dbt-core behavior. This would restore the existing workaround.

Alternatively, Fusion should provide a first-class way to dynamically select a Snowflake warehouse based on both `flags.FULL_REFRESH` **and** whether the model's relation already exists (`is_incremental()` / `load_relation(this)`), so users have a supported migration path away from the workaround.

## Steps to reproduce

1. Create a macro `use_warehouse` that runs `USE WAREHOUSE {{ warehouse }}` via `run_query`.
2. Use it in a model body guarded by `{% if not is_incremental() %}`.
3. Run `dbt run` on a relation that does not yet exist.
4. Observe that the model builds using the **default** warehouse, not the larger one specified by `use_warehouse`.
5. Compare to dbt-core behavior where the larger warehouse is correctly used.

## Is this a discrepancy between dbt Fusion Engine and dbt Core?

- [x] YES
- [ ] NO

## Impact

Customers with large incremental models that need a bigger warehouse for full-refresh or first-build runs are silently using the wrong (smaller) warehouse in Fusion, leading to slower runtimes or query failures. This was surfaced by customer **Norlys** during a Fusion migration.

## Version

Observed in current Fusion preview versions (confirmed via customer debug logs showing the `USE WAREHOUSE` executes but the model build reverts to default warehouse).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.