ClickHouse / ClickHouse/dbt-clickhouse
Feature parity for distributed and non-distributed materializations
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
With materializations split into non-distributed (standard) and distributed (non-standard) versions — i.e., `table` vs. `distributed_table` and `incremental` vs. `distributed_incremental` — we often encounter issues due to redundancy and divergence in their implementations. This has led to distributed materializations lacking full feature support or containing bugs in duplicated sections of the codebase.
**Some examples:**
- [Schema change validation](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/artifacts/resources/v1/config.py#L143) for contracted incremental models in `dbt-core` is only applied to `"incremental"` materializations. A related issue exists [here](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/task/run.py#L1135), which prevents microbatches from running when using the `distributed_incremental` strategy.
- Significant code duplication between the [incremental](https://github.com/ClickHouse/dbt-clickhouse/blob/main/dbt/include/clickhouse/macros/materializations/incremental/incremental.sql) and [distributed_incremental](https://github.com/ClickHouse/dbt-clickhouse/blob/main/dbt/include/clickhouse/macros/materializations/incremental/distributed_incremental.sql) materializations makes maintenance harder and introduces subtle bugs when switching between them.
**Suggested Solution**
Consolidate the distributed logic into the existing `incremental` and table `materializations`, controlled via a new model configuration flag (e.g., a boolean `is_distributed`). This would:
- Simplify the codebase by reducing redundancy.
- Ensure consistent feature support and behavior across both modes.
- Enable more comprehensive integration testing for both distributed and non-distributed materializations.
The existing `distributed_incremental` and `distributed_table` materializations could be retained as aliases (with deprecation warnings) for backwards compatibility.
Contributor guide
Research direction
Start by comparing the incremental and distributed_incremental materialization macros in dbt/include/clickhouse/macros/materializations/incremental/, then inspect the referenced dbt-core config.py and task/run.py locations. Done means distributed and non-distributed modes share feature behavior, integration coverage exists for both, and retained aliases emit deprecation warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- database
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100