dbt-labs / dbt-labs/dbt-adapters
[CT-1641] [CT-1622] [Bug] Insert overwrite incremental models using create or replace table
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug in dbt-core?
- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug
### Current Behavior
When I run an `incremental` table update using the `insert_overwrite` strategy and `require_partition_filter`=true and using `static partitions`, the table runs a `create or replace table` statement with a query syntax equivalent to (select * from table).
This returns an error: Cannot query over table XX without a filter over column(s) XX that can be used for partition elimination
I observed this for several models but others work fine. There is no difference in how these models are built.
### Expected Behavior
- Instead of `create or replace table` statement, i expect a merge statement to be ran (ie merge into table XX as DBT_INTERNAL_DEST using (query)) since this is an incremental table and **the target table exists/is not empty**
- Also, I expect the merge statement to be using the tmp table, but that's already raised here: https://github.com/dbt-labs/dbt-bigquery/issues/427
### Steps To Reproduce
I'll exclude the main body of the model for brevity but the config params are:
```
{{
config(
materialized='incremental',
incremental_strategy = "insert_overwrite",
cluster_by="country",
partition_by = {
"field": "report_date",
"data_type": "date",
"granularity": "day"
},
partitions = [DATE_SUB(CURRENT_DATE, INTERVAL 1 DAY), DATE_SUB(CURRENT_DATE, INTERVAL 2 DAY)],
require_partition_filter=true,
on_schema_change="sync_all_columns"
)
}}
```
after running the model, i looked at the logs
1. the 1st query created the tmp table correctly although its not used
2. the 2nd query ran a create or replace table instead of a merge
```
/* {"app": "dbt", "dbt_version": "1.1.2", "profile_name": "bigquery", "target_name": "prod", "node_id": "model_name"} */
create or replace table `model_name`
partition by report_date
cluster by country
OPTIONS(
description="""table description""",
labels=[('label')],
require_partition_filter=True
)
as (
select
(all columns)
from `model_name`
);
```
error is return as the create or replace statement is not using the partition filter, but it is also not correct in the first place as it should be a merge statement
### Relevant log output
_No response_
### Environment
```markdown
- OS: Big Sur 11.6
- Python: 3.8.0
- dbt:1.1.2
```
### Which database adapter are you using with dbt?
bigquery
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.