ClickHouse / ClickHouse/dbt-clickhouse

Duplicates in case if snapshot didn't finished successfully

Open
#600 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug
During the snapshot materialization we might receive duplicates and insert them into the target snapshot table.

It happens because we don't drop temporary tables before the snapshot. It means if we failed at some point when we already inserted data into the temporary table, but failed after (for example, by timeout) - on the next run we will have the same temporary table, but with already inserted data that can produce duplicates.

### dbt and/or ClickHouse server logs

### Configuration
#### Environment
* dbt version: 1.8.9
* dbt-clickhouse version: 1.8.9
* Python version: 3.10
* Operating system: Linux (K8s pod, triggered by Airflow)

#### ClickHouse server
* ClickHouse Server version: 25.8.11.66

Contributor guide

Open the contributing guide

Research direction

Start by tracing the snapshot materialization path that creates and populates temporary tables, then inspect what happens when a run fails after inserting data. Add coverage for rerunning after a failed materialization and verify that the target snapshot contains no duplicate rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.