[Feature] new configuration to run tests on only the "new" data for snapshots and incremental models
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Describe the feature
A more general solution for https://github.com/dbt-labs/dbt-core/issues/10236 and https://github.com/dbt-labs/dbt-core/issues/10864 and https://github.com/dbt-labs/dbt-snowflake/issues/1198
Folks want to be able to test _just_ their "new" data before it's inserted into their existing table. This is relevant for:
- snapshots
- incremental models
For example, you may want to test that your `unique_key` is actually unique for the "new" data. If that test fails _don't_ insert the "new" data into the existing table.
We should create a new [data test configuration](https://docs.getdbt.com/reference/data-test-configs) to allow you to configure a given data to _only_ run on the "new" data.
```
models:
- name: my_model
columns:
- name: id
data_tests:
- unique:
new_records_only: true # name/spec TBD
```
### Acceptance criteria:
- you can set this config on a data test for an incremental model
- you can set this config on a data test for a snapshot
- when set, the data test will only run on "new" data before insert/merge
- if data test fails, the subsequent insert/merge will be skipped
Contributor guide
Assessment
This issue has not been assessed yet.