dbt-labs / dbt-labs/dbt-adapters

[CT-1641] [CT-1622] [Bug] Insert overwrite incremental models using create or replace table

Open
#597 17 comments 1 reaction 0 assignees View on GitHub
feature:incremental pkg:dbt-bigquery type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

When I run an `incremental` table update using the `insert_overwrite` strategy and `require_partition_filter`=true and using `static partitions`, the table runs a `create or replace table` statement with a query syntax equivalent to (select * from table).

This returns an error: Cannot query over table XX without a filter over column(s) XX that can be used for partition elimination

I observed this for several models but others work fine. There is no difference in how these models are built.

### Expected Behavior

- Instead of `create or replace table` statement, i expect a merge statement to be ran (ie merge into table XX as DBT_INTERNAL_DEST using (query)) since this is an incremental table and **the target table exists/is not empty**
- Also, I expect the merge statement to be using the tmp table, but that's already raised here: https://github.com/dbt-labs/dbt-bigquery/issues/427

### Steps To Reproduce

I'll exclude the main body of the model for brevity but the config params are:

```
{{
config(
materialized='incremental',
incremental_strategy = "insert_overwrite",
cluster_by="country",
partition_by = {
"field": "report_date",
"data_type": "date",
"granularity": "day"
},
partitions = [DATE_SUB(CURRENT_DATE, INTERVAL 1 DAY), DATE_SUB(CURRENT_DATE, INTERVAL 2 DAY)],
require_partition_filter=true,
on_schema_change="sync_all_columns"
)
}}
```
after running the model, i looked at the logs
1. the 1st query created the tmp table correctly although its not used
2. the 2nd query ran a create or replace table instead of a merge
```

/* {"app": "dbt", "dbt_version": "1.1.2", "profile_name": "bigquery", "target_name": "prod", "node_id": "model_name"} */

create or replace table `model_name`
partition by report_date
cluster by country
OPTIONS(
description="""table description""",

labels=[('label')],

require_partition_filter=True
)
as (

select
(all columns)
from `model_name`
);
```
error is return as the create or replace statement is not using the partition filter, but it is also not correct in the first place as it should be a merge statement

### Relevant log output

_No response_

### Environment

```markdown
- OS: Big Sur 11.6
- Python: 3.8.0
- dbt:1.1.2
```

### Which database adapter are you using with dbt?

bigquery

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.