ClickHouse / ClickHouse/dbt-clickhouse

insert_overwrite strategy can miss data on node failover

Open
#418 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

When using the `insert_overwrite` strategy, it's possible that not all the statements run on the same node, leading to some data not being propagated.

The sequence of operation in https://github.com/ClickHouse/dbt-clickhouse/blob/57db2b2c87243273c44a1e8abd96a17ae3ebf806/dbt/include/clickhouse/macros/materializations/incremental/incremental.sql is:
1. Create new data table
2. Insert-select to new data table
3. Create temporary table
4. Insert-select into temporary table from new data table
5. Find partition_id in temporary table by querying system.parts
6. `ALTER TABLE main_table REPLACE PARTITION ID partition_id FROM `

If the node changes between steps 5 and 6 because of a failover, the REPLACE PARTITION can execute without moving any data since not all nodes will immediately know about all the new data inserted in step 4.

ClickHouse will not throw an error when replacing a partition with no parts (not a good default and something we should fix as well) and in the logs this will show up with a log line containing `REPLACE cloning 0 parts for partition` in `system.text_log` for the REPLACE PARTITION query.

One way to address this would be to run a `SYSTEM SYNC REPLICA ON CLUSTER . LIGHTWEIGHT` after step 4 to make sure all nodes have all the information about the new parts.

Contributor guide

Open the contributing guide

Research direction

Start in dbt/include/clickhouse/macros/materializations/incremental/incremental.sql and trace the insert_overwrite sequence around the temporary-table insert, system.parts query, and REPLACE PARTITION. Reproduce or inspect a node failover between steps 5 and 6, using system.text_log to check for "REPLACE cloning 0 parts for partition"; done means the replacement cannot silently miss newly inserted data.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.