dbt-labs / dbt-labs/docs.getdbt.com
The guide "Incremental models in-depth" is not valid for BigQuery with partitioned tables
- Dominant language
- JavaScript
- Stars
- 215
- Forks
- 1.2k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 130
Description
### Contributions
- [x] I have read the contribution docs, and understand what's expected of me.
### Link to the page on docs.getdbt.com requiring updates
https://docs.getdbt.com/best-practices/materializations/4-incremental-models
### What part(s) of the page would you like to see updated?
This code
```
{% if is_incremental() %}
where
updated_at > (select max(updated_at) from {{ this }})
{% endif %}
```
won't filter on partitions in BigQuery and can be extremely expensive 🤑 . [See docs here on partition pruning in BQ](https://cloud.google.com/bigquery/docs/querying-partitioned-tables#use_a_constant_filter_expression).
For BQ, with partitioned tables, we should calculate the value first with a `run_query()` and then use it.
Example:
```
{%- set max_date = dbt_utils.get_single_value("select max(updated_at) from {{ this }}", default="'2020-01-01'") -%}
{% if is_incremental() %}
where
updated_at > {{ max_date }}
{% endif %}
```
### Additional information
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.