dbt-labs / dbt-labs/dbt

[Enhancement] Support SQL expressions within `check_cols` for snapshots

Open
#11,507 3 comments 1 reaction 0 assignees View on GitHub
engine:v1 snapshots type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [x] I believe this is a new bug in dbt-core
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

Hello,

While using dbt snapshot, with the following .yaml file:

```
snapshots:
- name: snapshot_test
relation: source('david_playground', 'table_with_nested_field')
config:
strategy: check
unique_key: id
check_cols:
- repeated_string
dbt_valid_to_current: "TIMESTAMP(DATE('9999-12-31'))"
hard_deletes: invalidate
```

It raises an error, because ARRAY (repeated field) are not comparable. This behaviour is good.

Then, I try to use `TO_JSON_STRING(repeated_string)` instead of `repeated_string` to have a proper SQL comparison.

Instead of trying to make the comparison, DBT release the following SQL for the update statement part:
```
updates as (
select
'update' as dbt_change_type,
source_data.*,
snapshotted_data.dbt_scd_id
from updates_source_data as source_data
join snapshotted_data
on
snapshotted_data.dbt_unique_key = source_data.dbt_unique_key
where ((TRUE))
)
```

Instead of trying to make the condition with TO_JSON_STRING(repeated_string), it just put TRUE.

No error is raised, and the snapshot will consider all rows of my source like updates, even if there is no update.
This is not expected in a snapshot in which I want to track only changes.

### Expected Behavior

Solution 1 : generate the condition with `TO_JSON_STRING(snapshotted_data.repeated_string) != TO_JSON_STRING(source_data.repeated_string)`

or

Solution 2 : raise an error to say SQL Function are not usable here

### Steps To Reproduce

1. In Bigquery, create a source table with id and repeated value
2. Run `dbt snapshot`
3. It's not working with the repeated value
4. Try to use `TO_JSON_STRING(your_repeated_field)` in `check_cols` of the yaml fille
5. Run `dbt snapshot` again
6. It's working incorrectly

### Relevant log output

```shell

```

### Environment

```markdown
- OS: MacOS
- Python: 3.11
- dbt: 1.9.4
```

### Which database adapter are you using with dbt?

bigquery

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.