[Feature] New snapshot config to validate uniqueness before merge
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
Originally posted by @dbeatty10 in https://github.com/dbt-labs/dbt-core/issues/6089#issuecomment-1409369785:
> @kjstultz thanks again for creating this issue! Appreciate that you invested the time at Coalesce in NOLA to figure out how to reproduce this tricky situation.
>
> Reading the discussion between you and @jtcohen6, it sounds like the feature discussed thus far would look like this:
> - Add a new snapshot config parameter `validate_uniqueness` (default to `false`)
> - When `validate_uniqueness` is true, execute a uniqueness test after creating the staging table and before running the `merge`
>
> Although this proposed feature wouldn't fix a snapshot table that _already_ has duplicates, it would prevent duplicates from being added in the first place! Feels like a win to me, especially with it being configurable for those that (somehow) **know** that their "unique" key is truly unique.
---
Notes:
- Would be slow in some cases, so want to make sure this is configurable
- But valuable to make sure snapshot is accurate
Contributor guide
Assessment
This issue has not been assessed yet.