dbt-labs / dbt-labs/dbt

[CT-3211] [Feature] Globally unique temporary tables for incremental models

Open
#8,837 3 comments 2 reactions 0 assignees View on GitHub
engine:v1 help_wanted type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

## Feature

We would like temporary tables used by dbt to merge into incremental models to either:

- Be named in a globally unique fashion - e.g. using a UUID suffix
- Support specifying the target for the temporary table so that the end-user can specify a jinja-template for the suffix/name of the temporary table.

## Context

We have a multitenant architecture where each tenant has its own separate materialization of each model (separated in different datasets in BigQuery).
In our latest feature, we want to provide industry wide analytics that compare our different tenants to each other.
In order to achieve this, we created an industry-wide incremental model with `insert_overwrite` strategy, partitioned by a unique integer for each of our tenants. Exact config block:

```
config(
materialized='incremental',
incremental_strategy='insert_overwrite',
partition_by={
'field': 'client_int_partition_key',
'data_type': 'int64',
'range': {
'start': 1,
'end': 4000,
'interval': 1
}
},
)
```

Conceptually, this is great as it allows us to have all our tenants write to the same centralized table even if their company's data is located in their own tenant-scoped datasets.
However, in practice an issue arise when >=2 tenants attempt to write to the industry-scoped incremental model, as the incremental macro implementation creates a temporary table, enacts the merge to the incremental model, and then deletes the temporary table. The >=2 tenants end up conflicting with each other either by deleting the table before the other's operations are done or by truncating the data of the first tenant with the data of the next one.

The exact error produced by dbt is the following:

```
INFO - 23:50:46 Database Error in model {model_name} (path/to/model_name.sql)
INFO - 23:50:46 Destination deleted/expired during execution: {project}:{dataset}.{model_name}__dbt_tmp. at [20:3]
INFO - 23:50:46 compiled SQL at {target-path}/run/path/to/{model_name}.sql
```

Hence the suggested solution is to change the suffix `__dbt_tmp` to `__dbt_tmp_{GUID/user provided suffix}`.
I think the GUID version would be way simpler to implement, and not break Dependency Inversion Principle (SOLID).

### Describe alternatives you've considered

For the time being, we'll be using [Airflow's Pools](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html) to prevent parallelism in execution of the incremental insertions. That way there won't be any conflict between the various tenants, they will run their incremental insertions sequentially.

EDIT: After implementing the alternative, it is confirmed approach solves the issue.

### Who will this benefit?

Anyone who may have multiple actors/processes appending to a model concurrently.

### Are you interested in contributing this feature?

Yes, I think that would be doable if we choose on the GUID solution.
All adapters must have some form of GUID I can use, so the feature could be simple to implement?

### Anything else?

:)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.