ClickHouse / ClickHouse/dbt-clickhouse
Microbatches can't be executed concurrently.
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
We have a few issues related to this:
- It's not included as one of the supported capabilities, so this is actually not working [impl.py#L76-L80](https://github.com/ClickHouse/dbt-clickhouse/blob/v1.10.0/dbt/adapters/clickhouse/impl.py#L76-L80)
- Also the microbatch internally uses `clickhouse__incremental_delete_insert` that creates a staging table. The table is created with the same name per batch execution, so concurrent executions may collide.
We mention that concurrency is possible in the docs. We need to check if this is true because it looks like that setting is just silently ignored.
### Expected behaviour
Concurrent executions doesn't collide. To fix it we should probably create these tables appending the `model.batch.id` which is supposed to be different per batch. We need to verify it.
Contributor guide
Research direction
Start with dbt/adapters/clickhouse/impl.py#L76-L80 and trace the microbatch path through clickhouse__incremental_delete_insert. Check the documentation claim about concurrent execution and verify whether model.batch.id is available for staging-table names. Done means concurrent microbatches are supported without staging-table collisions, or the documentation accurately reflects the limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100