"unique_key" is a confusing name for config, does different things for snapshots vs. incremental models
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
## Problem
`unique_key` is a config available for both snapshots and incremental models (but [handles each slightly differently](https://github.com/dbt-labs/docs.getdbt.com/issues/4703))
### `unique_key` for incremental models
[unique_key](https://docs.getdbt.com/docs/build/incremental-models#defining-a-unique-key-optional) (for incremental models) - the field used for determining which rows to “update” in the MERGE statement.
While in theory this is supposed to be equivalent to your primary key, [folks don’t always use it that way](https://github.com/dbt-labs/docs.getdbt.com/issues/4355).
#### Let's look at an example of when `unique_key` is equivalent to the primary key.
Let’s say I have an incremental model:
```sql
{{
config(
materialized='incremental',
unique_key='event_id'
)
}}
with source as (
select * from {{ ref('stg_events') }}
{% if is_incremental() %}
-- this filter will only be applied on an incremental run
where _etl_loaded_at > (select {{ dbt.dateadd("hour", -3, "max(_etl_loaded_at)") }} from {{ this }})
{% endif %}
)
select * from source
```
In this case `event_id` is my primary key.
Let’s say my table currently has a record for `event_id` = 1:
event_id | event_type | user_name | _etl_loaded_at
-- | -- | -- | --
1 | new sign up | grack goheen | 2024-02-02T16:22:57+00:00
If I execute this model during an incremental run with new data for event_id = 1, my new data will be merged into this table, so that I maintain a single record for each event_id:
event_id | event_type | user_name | _etl_loaded_at
-- | -- | -- | --
1 | new sign up | grace goheen | 2024-02-02T16:23:57+00:00
#### Let's look at an example of when `unique_key` is NOT necessarily equivalent to the primary key - `unique_key` config really is just the “upsert/merge key” (could be primary key, could be date partition, etc.)
Let’s say I have an incremental model:
```sql
{{
config(
materialized='incremental',
unique_key='date_day',
merge_strategy='delete+insert'
)
}}
with source as (
select * from {{ ref('stg_events') }}
{% if is_incremental() %}
-- this filter will only be applied on an incremental run
where date_day > (select {{ dbt.dateadd("day", -3, "max(date_day)") }} from {{ this }})
{% endif %}
)
select * from source
```
Let’s say my table currently has a records for `date_day = 02-02-2024`:
event_id | event_type | user_name | date_day | _etl_loaded_at
-- | -- | -- | -- | --
1 | new sign up | grace goheen | 02-02-2024 | 2024-02-02T16:23:57+00:00
2 | new sign up | dave connors | 02-02-2024 | 2024-02-02T16:23:58+00:00
3 | unsubscribe | grace goheen | 02-02-2024 | 2024-02-02T18:23:57+00:00
If I execute this model during an incremental run with less data for `date_day = 02-02-2024`, my old data for `02-02-2024` be deleted from this table and the new data will be inserted:
event_id | event_type | user_name | date_day | _etl_loaded_at
-- | -- | -- | -- | --
1 | new sign up | grace goheen | 02-02-2024 | 2024-02-02T16:23:57+00:00
2 | unsubscribe | grace goheen | 02-02-2024 | 2024-02-02T18:23:57+00:00
In this case `date_day` is NOT my `primary key` (`event_id` is), even though I’m using it for my `unique_key` config.
### `unique_key` for snapshots
[unique_key](https://docs.getdbt.com/reference/resource-configs/unique_key) (for snapshots) - the field used to match records between a result set and an existing snapshot, so that changes can be captured correctly
- this is unique for the snapshot query result, not the resulting snapshot table
- technically, Jinja-SQL snapshots `{% snapshot my_snapshot %}` can define a whole complex gnarly transformation query, and the configured `unique_key` is a column of the outputted dataset — but that's not best practice!
## Acceptance criteria
tbd
Contributor guide
Assessment
This issue has not been assessed yet.