dbt-labs / dbt-labs/dbt

[Feature] Safer incrementals/snapshots (don't allow `create or replace ...`)

Open
#15,953 2 comments 1 reaction 0 assignees View on GitHub
area:adapters engine:v1 engine:v2 status:triage type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Which version of dbt is this feature for?

dbt v2.x (Core or Fusion)

### Describe the feature

From here: https://github.com/jeremyyeo/the-hitchhikers-guide-to-dbt/tree/main/preventing-incremental-models-from-being-recreated

The zen of dbt is that your data transformations should be idempotent (besides snapshots) - i.e. you should be able to `dbt build --full-refresh` and get all your models into the right state at any time.

Note that an incremental model / snapshot will be created from scratch (`create or replace ...`) if the relation/adapter cache that populates at the start of the invocation does not contain the table - i.e. to dbt, the table does not exist. This may happen because:
1. The table literally does not exist / no access to the table / the dwh did not return the right information (there may be a bug with the information schema - which is out of dbt control).
2. There is a bug with the adapter cache.

Depending on the nature of the data (volume, is the transform actually idempotent) - this can be a very bad outcome (less bad: rack up compute cost rebuilding incremental from scratch, worse case: lost of data). dbt should implement a mechanism to prevent `create or replace ...` statements. Starting an issue to track this / discuss options.

### Describe alternatives you've considered

1. Enterprising analytics engineers can rewrite ootb materializations - example shared in linked github above.

### Who will this benefit?

Users with non-idempotent transforms that are "critical".

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.