aws-samples / aws-samples/dbt-glue
Concurrency Issue with Iceberg Incremental Models' Temporary Table Creation
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the bug
When running multiple **concurrent incremental Iceberg** models, Some of them fail with the following error:
`Py4JJavaError: An error occurred while calling o244.sql. java.lang.IllegalStateException: Table UUID does not match: current=831376b4-a28d-4fd5-a748-d167244ed290 != refreshed=06d6706e-e54a-4878-b410-9a7126214cfb`
According to the logs, the SQL that fails is the one responsible for creating the temporary Iceberg table for staging the incremental data (before loading it to the target table).
While the data loading phase (using `INSERT INTO` or `MERGE`, etc. according to the selected strategy) supports concurrent runs via Iceberg's built-in features, the creation of the temporary Iceberg table apparently does not.
Additionally, even if the temporary table creation did not fail, the current design of the temporary table (that is meant to stage data for a specific run) would lead to data loss/failures in concurrent scenarios. If multiple runs were to proceed, the temporary table would only stage data for the last successful run (can be of the same model or of another model that has a different schema). Consequently, all concurrent runs would attempt to load the same data to the target tables, leading to incorrect/missing data, or failures if different models within the same database are ran concurrently.
### Steps To Reproduce
1. Configure a dbt project to use dbt-glue
2. create an incremental model with file_format='iceberg'
3. Initiate multiple `dbt run --select your_iceberg_incremental_model` commands concurrently. (e.g., using a CI/CD pipeline that triggers multiple runs simultaneously, or by manually executing the command in multiple terminals at roughly the same time).
4. Observe the failure of one or more of the concurrent runs during the temporary table creation phase, with the `Py4JJavaError` mentioned above.
5. (Hypothetically, if the error did not occur) Observe that all concurrent runs would attempt to load the same data, leading to data loss in the target table.
### Expected behavior
Ideally, each concurrent incremental materialization run should create and utilize its own isolated temporary table for staging incremental data. This would ensure that each run processes its unique set of incremental data without interference from other concurrent processes. Once the data is staged in separate temporary tables, Iceberg's inherent concurrency handling capabilities for write operations would then manage the concurrent writes from these distinct temporary tables to the final target table.
### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.9.8
- latest: 1.10.3 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- glue: 1.9.4 - Up to date!
- spark: 1.9.2 - Up to date!
```
**The operating system you're using:**
macOS
**The output of `python --version`:**
3.10.10
### Additional context
We have implemented a workaround locally by modifying the `glue__make_temp_relation` macro responsible for generating the temporary table name. We have appended a unique UUID to the temporary table name so that each concurrent run creates a distinct temporary table. This approach is working fine now in our processes.
```
{% macro glue__make_temp_relation(base_relation, suffix) %}
{% call statement('unique_id', fetch_result=True) -%}
select regexp_replace(uuid(), '-', '_') as unique_str
{% endcall %}
{% set unique_id = load_result('unique_id')['data'][0][0] %}
{% set tmp_identifier = base_relation.identifier ~ suffix ~ unique_id %}
{% set tmp_relation = base_relation.incorporate(path={"schema": base_relation.schema, "identifier": tmp_identifier}) -%}
{% do return(tmp_relation) %}
{% endmacro %}
```
Contributor guide
Research direction
Start with the glue__make_temp_relation macro and trace where it is used during concurrent incremental Iceberg materializations. Reproduce the parallel dbt runs described in the issue, then verify that each run stages data in an isolated temporary table without UUID mismatches, data collisions, or failures when schemas differ.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100