[Feature] Safer incrementals/snapshots (don't allow `create or replace ...`)
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Which version of dbt is this feature for?
dbt v2.x (Core or Fusion)
### Describe the feature
From here: https://github.com/jeremyyeo/the-hitchhikers-guide-to-dbt/tree/main/preventing-incremental-models-from-being-recreated
The zen of dbt is that your data transformations should be idempotent (besides snapshots) - i.e. you should be able to `dbt build --full-refresh` and get all your models into the right state at any time.
Note that an incremental model / snapshot will be created from scratch (`create or replace ...`) if the relation/adapter cache that populates at the start of the invocation does not contain the table - i.e. to dbt, the table does not exist. This may happen because:
1. The table literally does not exist / no access to the table / the dwh did not return the right information (there may be a bug with the information schema - which is out of dbt control).
2. There is a bug with the adapter cache.
Depending on the nature of the data (volume, is the transform actually idempotent) - this can be a very bad outcome (less bad: rack up compute cost rebuilding incremental from scratch, worse case: lost of data). dbt should implement a mechanism to prevent `create or replace ...` statements. Starting an issue to track this / discuss options.
### Describe alternatives you've considered
1. Enterprising analytics engineers can rewrite ootb materializations - example shared in linked github above.
### Who will this benefit?
Users with non-idempotent transforms that are "critical".
### Are you interested in contributing this feature?
_No response_
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.