ClickHouse / ClickHouse/dbt-clickhouse
insert_overwrite strategy can miss data on node failover
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
When using the `insert_overwrite` strategy, it's possible that not all the statements run on the same node, leading to some data not being propagated.
The sequence of operation in https://github.com/ClickHouse/dbt-clickhouse/blob/57db2b2c87243273c44a1e8abd96a17ae3ebf806/dbt/include/clickhouse/macros/materializations/incremental/incremental.sql is:
1. Create new data table
2. Insert-select to new data table
3. Create temporary table
4. Insert-select into temporary table from new data table
5. Find partition_id in temporary table by querying system.parts
6. `ALTER TABLE main_table REPLACE PARTITION ID partition_id FROM `
If the node changes between steps 5 and 6 because of a failover, the REPLACE PARTITION can execute without moving any data since not all nodes will immediately know about all the new data inserted in step 4.
ClickHouse will not throw an error when replacing a partition with no parts (not a good default and something we should fix as well) and in the logs this will show up with a log line containing `REPLACE cloning 0 parts for partition` in `system.text_log` for the REPLACE PARTITION query.
One way to address this would be to run a `SYSTEM SYNC REPLICA ON CLUSTER . LIGHTWEIGHT` after step 4 to make sure all nodes have all the information about the new parts.
Contributor guide
Research direction
Start in dbt/include/clickhouse/macros/materializations/incremental/incremental.sql and trace the insert_overwrite sequence around the temporary-table insert, system.parts query, and REPLACE PARTITION. Reproduce or inspect a node failover between steps 5 and 6, using system.text_log to check for "REPLACE cloning 0 parts for partition"; done means the replacement cannot silently miss newly inserted data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100