ClickHouse / ClickHouse/dbt-clickhouse

Feature parity for distributed and non-distributed materializations

Open
#439 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

With materializations split into non-distributed (standard) and distributed (non-standard) versions — i.e., `table` vs. `distributed_table` and `incremental` vs. `distributed_incremental` — we often encounter issues due to redundancy and divergence in their implementations. This has led to distributed materializations lacking full feature support or containing bugs in duplicated sections of the codebase.

**Some examples:**

- [Schema change validation](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/artifacts/resources/v1/config.py#L143) for contracted incremental models in `dbt-core` is only applied to `"incremental"` materializations. A related issue exists [here](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/task/run.py#L1135), which prevents microbatches from running when using the `distributed_incremental` strategy.

- Significant code duplication between the [incremental](https://github.com/ClickHouse/dbt-clickhouse/blob/main/dbt/include/clickhouse/macros/materializations/incremental/incremental.sql) and [distributed_incremental](https://github.com/ClickHouse/dbt-clickhouse/blob/main/dbt/include/clickhouse/macros/materializations/incremental/distributed_incremental.sql) materializations makes maintenance harder and introduces subtle bugs when switching between them.

**Suggested Solution**

Consolidate the distributed logic into the existing `incremental` and table `materializations`, controlled via a new model configuration flag (e.g., a boolean `is_distributed`). This would:

- Simplify the codebase by reducing redundancy.
- Ensure consistent feature support and behavior across both modes.
- Enable more comprehensive integration testing for both distributed and non-distributed materializations.

The existing `distributed_incremental` and `distributed_table` materializations could be retained as aliases (with deprecation warnings) for backwards compatibility.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the incremental and distributed_incremental materialization macros in dbt/include/clickhouse/macros/materializations/incremental/, then inspect the referenced dbt-core config.py and task/run.py locations. Done means distributed and non-distributed modes share feature behavior, integration coverage exists for both, and retained aliases emit deprecation warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
database
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.