dbt-labs / dbt-labs/dbt

dbt snapshots to handle append tables

Open
#3,878 6 comments 5 reactions 0 assignees View on GitHub
engine:v1 snapshots type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Describe the feature
dbt snapshot feature to handle scd2 changes when a give id exists several times (data is appended to the source table, as opposed to overwritten). Currenty, dbt snapshots assume that:
- There is one row per unique id
- In the source data, the original row gets overwritten when a record is modified (and not appended).

### Describe alternatives you've considered
The way our data lake works (dbt reads from the data lake and writes to our data warehouse) - those are two distinct databases in Snowflake.
We get a FULL copy of the prod table every day, clustered by date. Thus, we have a full table every day. The delta between day and day-1 is what we need for scd2. The only way I’ve made it work so far is: that snapshots pull from the latest version of the table in prod.
Problem: if we lose a day or two, we need to re-run for the missing days, which I’ve made work by passing a variable like `dbt snapshot --var ref_date = 'my date'`
Sadly, snapshots in dbt don’t allow to iterate over an array of dates

### Additional context
It's not db specific but snowflake does throw this `ERROR_ON_NONDETERMINISTIC_MERGE` if an id exists several times.
When set to false, Snowflake doesn't throw any error but does not handle scd2 changes properly. (see dates below):
```
ID | Updated_at | Valid from | Valid to
1 | 2020-12-15 | 2020-12-15 | 2021-08-19
1 | 2021-07-13 | 2021-07-13 | 2021-08-19
1 | 2021-08-19 | 2021-08-19 | NULL
```

### Who will this benefit?
Anyone who wishes to have scd2 snapshots derived from append tables or create a scd2 table off off an incremental table where an id can exists several time.

### Are you interested in contributing this feature?
Yes. I am happy to look into the SQL, what needs to be done, update the macro.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.