ClickHouse / ClickHouse/dbt-clickhouse

Microbatches can't be executed concurrently.

Open
#633 3 comments 4 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug
We have a few issues related to this:
- It's not included as one of the supported capabilities, so this is actually not working [impl.py#L76-L80](https://github.com/ClickHouse/dbt-clickhouse/blob/v1.10.0/dbt/adapters/clickhouse/impl.py#L76-L80)
- Also the microbatch internally uses `clickhouse__incremental_delete_insert` that creates a staging table. The table is created with the same name per batch execution, so concurrent executions may collide.

We mention that concurrency is possible in the docs. We need to check if this is true because it looks like that setting is just silently ignored.

### Expected behaviour
Concurrent executions doesn't collide. To fix it we should probably create these tables appending the `model.batch.id` which is supposed to be different per batch. We need to verify it.

Contributor guide

Open the contributing guide

Research direction

Start with dbt/adapters/clickhouse/impl.py#L76-L80 and trace the microbatch path through clickhouse__incremental_delete_insert. Check the documentation claim about concurrent execution and verify whether model.batch.id is available for staging-table names. Done means concurrent microbatches are supported without staging-table collisions, or the documentation accurately reflects the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.