ClickHouse / ClickHouse/dbt-clickhouse
Duplicates in case if snapshot didn't finished successfully
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
During the snapshot materialization we might receive duplicates and insert them into the target snapshot table.
It happens because we don't drop temporary tables before the snapshot. It means if we failed at some point when we already inserted data into the temporary table, but failed after (for example, by timeout) - on the next run we will have the same temporary table, but with already inserted data that can produce duplicates.
### dbt and/or ClickHouse server logs
### Configuration
#### Environment
* dbt version: 1.8.9
* dbt-clickhouse version: 1.8.9
* Python version: 3.10
* Operating system: Linux (K8s pod, triggered by Airflow)
#### ClickHouse server
* ClickHouse Server version: 25.8.11.66
Contributor guide
Research direction
Start by tracing the snapshot materialization path that creates and populates temporary tables, then inspect what happens when a run fails after inserting data. Add coverage for rerunning after a failed materialization and verify that the target snapshot contains no duplicate rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100