dbt-labs / dbt-labs/docs.getdbt.com

The guide "Incremental models in-depth" is not valid for BigQuery with partitioned tables

Open
#4,955 2 comments 0 reactions 0 assignees View on GitHub
content improvement
Dominant language
JavaScript
Stars
215
Forks
1.2k
Avg merge
1d 15h
Merged PRs (30d)
130

Description

### Contributions

- [x] I have read the contribution docs, and understand what's expected of me.

### Link to the page on docs.getdbt.com requiring updates

https://docs.getdbt.com/best-practices/materializations/4-incremental-models

### What part(s) of the page would you like to see updated?

This code

```
{% if is_incremental() %}

where
updated_at > (select max(updated_at) from {{ this }})

{% endif %}
```

won't filter on partitions in BigQuery and can be extremely expensive 🤑 . [See docs here on partition pruning in BQ](https://cloud.google.com/bigquery/docs/querying-partitioned-tables#use_a_constant_filter_expression).

For BQ, with partitioned tables, we should calculate the value first with a `run_query()` and then use it.

Example:
```
{%- set max_date = dbt_utils.get_single_value("select max(updated_at) from {{ this }}", default="'2020-01-01'") -%}

{% if is_incremental() %}

where
updated_at > {{ max_date }}

{% endif %}
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.