dbt-labs / dbt-labs/dbt

[Feature] New snapshot config to validate uniqueness before merge

Open
#10,236 1 comment 1 reaction 0 assignees View on GitHub
engine:v1 snapshots status:triage type:docs type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

Originally posted by @dbeatty10 in https://github.com/dbt-labs/dbt-core/issues/6089#issuecomment-1409369785:

> @kjstultz thanks again for creating this issue! Appreciate that you invested the time at Coalesce in NOLA to figure out how to reproduce this tricky situation.
>
> Reading the discussion between you and @jtcohen6, it sounds like the feature discussed thus far would look like this:
> - Add a new snapshot config parameter `validate_uniqueness` (default to `false`)
> - When `validate_uniqueness` is true, execute a uniqueness test after creating the staging table and before running the `merge`
>
> Although this proposed feature wouldn't fix a snapshot table that _already_ has duplicates, it would prevent duplicates from being added in the first place! Feels like a win to me, especially with it being configurable for those that (somehow) **know** that their "unique" key is truly unique.

---

Notes:
- Would be slow in some cases, so want to make sure this is configurable
- But valuable to make sure snapshot is accurate

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.