dbt snapshots to handle append tables
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Describe the feature
dbt snapshot feature to handle scd2 changes when a give id exists several times (data is appended to the source table, as opposed to overwritten). Currenty, dbt snapshots assume that:
- There is one row per unique id
- In the source data, the original row gets overwritten when a record is modified (and not appended).
### Describe alternatives you've considered
The way our data lake works (dbt reads from the data lake and writes to our data warehouse) - those are two distinct databases in Snowflake.
We get a FULL copy of the prod table every day, clustered by date. Thus, we have a full table every day. The delta between day and day-1 is what we need for scd2. The only way I’ve made it work so far is: that snapshots pull from the latest version of the table in prod.
Problem: if we lose a day or two, we need to re-run for the missing days, which I’ve made work by passing a variable like `dbt snapshot --var ref_date = 'my date'`
Sadly, snapshots in dbt don’t allow to iterate over an array of dates
### Additional context
It's not db specific but snowflake does throw this `ERROR_ON_NONDETERMINISTIC_MERGE` if an id exists several times.
When set to false, Snowflake doesn't throw any error but does not handle scd2 changes properly. (see dates below):
```
ID | Updated_at | Valid from | Valid to
1 | 2020-12-15 | 2020-12-15 | 2021-08-19
1 | 2021-07-13 | 2021-07-13 | 2021-08-19
1 | 2021-08-19 | 2021-08-19 | NULL
```
### Who will this benefit?
Anyone who wishes to have scd2 snapshots derived from append tables or create a scd2 table off off an incremental table where an id can exists several time.
### Are you interested in contributing this feature?
Yes. I am happy to look into the SQL, what needs to be done, update the macro.
Contributor guide
Assessment
This issue has not been assessed yet.