dbt-labs / dbt-labs/dbt

[Feature] new configuration to run tests on only the "new" data for snapshots and incremental models

Open
#10,877 3 comments 6 reactions 0 assignees View on GitHub
cardboard_cut engine:v1 type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Describe the feature

A more general solution for https://github.com/dbt-labs/dbt-core/issues/10236 and https://github.com/dbt-labs/dbt-core/issues/10864 and https://github.com/dbt-labs/dbt-snowflake/issues/1198

Folks want to be able to test _just_ their "new" data before it's inserted into their existing table. This is relevant for:
- snapshots
- incremental models

For example, you may want to test that your `unique_key` is actually unique for the "new" data. If that test fails _don't_ insert the "new" data into the existing table.

We should create a new [data test configuration](https://docs.getdbt.com/reference/data-test-configs) to allow you to configure a given data to _only_ run on the "new" data.

```
models:
- name: my_model
columns:
- name: id
data_tests:
- unique:
new_records_only: true # name/spec TBD
```

### Acceptance criteria:
- you can set this config on a data test for an incremental model
- you can set this config on a data test for a snapshot
- when set, the data test will only run on "new" data before insert/merge
- if data test fails, the subsequent insert/merge will be skipped

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.