aws-samples / aws-samples/dbt-glue
Create `tmp` tables with generated unique name.
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the feature
A clear and concise description of what you want to happen.
With the following configuration:
```
{{
config(
materialized = "incremental",
file_format = "parquet",
partition_by = ["created_date"],
incremental_strategy = "insert_overwrite",
)
}}
```
I expected that we could run dbt run -s my_model in parallel with multiple different created_date values passed to `vars`, for instance, --vars "created_date: 2023-01-01", --vars "created_date: 2023-01-02", ...
However, with the current version 1.7.0, although it seems impossible, dbt-glue will raise an error:
```
AnalysisException: Table or view 'my_table_tmp' not found in database 'my_db'
```
For example, if we first run `dbt run -s my_model --vars "created_date: 2023-01-01"`, dbt-glue will create `my_model_tmp`. Then, if we subsequently run `dbt run -s my_model --vars "created_date: 2023-01-02"`, dbt-glue will attempt to drop `my_model_tmp` first. This leads to the first run failing as described above.
### Describe alternatives you've considered
For each run, need to create `tmp` table with unique name.
### Additional context
Please include any other relevant context here.
### Who will this benefit?
Help run `incremental` model with `insert_overwrite` strategy in parrallel.
### Are you interested in contributing this feature?
I am new to dbt-glue, so I hope for a mentor to guide me through it.
Contributor guide
Research direction
The issue names no files or tests. Start by tracing dbt-glue's incremental insert_overwrite execution and temporary-table lifecycle, then examine how concurrent runs using different created_date vars interact. Done means parallel runs create distinct temporary tables without dropping or interfering with one another.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100