aws-samples / aws-samples/dbt-glue

Create `tmp` tables with generated unique name.

Open
#289 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the feature
A clear and concise description of what you want to happen.
With the following configuration:
```
{{
config(
materialized = "incremental",
file_format = "parquet",
partition_by = ["created_date"],
incremental_strategy = "insert_overwrite",
)
}}
```
I expected that we could run dbt run -s my_model in parallel with multiple different created_date values passed to `vars`, for instance, --vars "created_date: 2023-01-01", --vars "created_date: 2023-01-02", ...
However, with the current version 1.7.0, although it seems impossible, dbt-glue will raise an error:
```
AnalysisException: Table or view 'my_table_tmp' not found in database 'my_db'
```
For example, if we first run `dbt run -s my_model --vars "created_date: 2023-01-01"`, dbt-glue will create `my_model_tmp`. Then, if we subsequently run `dbt run -s my_model --vars "created_date: 2023-01-02"`, dbt-glue will attempt to drop `my_model_tmp` first. This leads to the first run failing as described above.

### Describe alternatives you've considered
For each run, need to create `tmp` table with unique name.

### Additional context
Please include any other relevant context here.

### Who will this benefit?
Help run `incremental` model with `insert_overwrite` strategy in parrallel.

### Are you interested in contributing this feature?
I am new to dbt-glue, so I hope for a mentor to guide me through it.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by tracing dbt-glue's incremental insert_overwrite execution and temporary-table lifecycle, then examine how concurrent runs using different created_date vars interact. Done means parallel runs create distinct temporary tables without dropping or interfering with one another.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.