ClickHouse / ClickHouse/dbt-clickhouse

dbt Core 2.0 parity: Materializations `distributed_table` / `distributed_incremental`

Open
#707 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

Sub-issue of #660 — feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion versus the Python `dbt-clickhouse` adapter.

Covers the `distributed_table` and `distributed_incremental` materializations. Both macros are merged at near-v1 parity, **but no distributed model can run**: cluster resolution never succeeds on main, so both materializations abort at compile time with `To use distributed materialization cluster setting in dbt profile must be set`, even when the profile sets `cluster:`. The cluster machinery they depend on is tracked in #711; the distributed-only `sharding_key` config key is also missing. Both materializations are self-managed-cluster features; ClickHouse Cloud users are not affected by this gap.

**Status legend:** ✅ implemented — merged and working on dbt Core 2.0 `origin/main` · 🟡 partial — partly working on `origin/main` (Notes say what works vs. what fails) · 🚧 in progress — implemented in an open PR (linked in Notes) · 🔴 not started — not working anywhere · Combined status "A-🚧->B": the feature is at A on origin/main today; an open PR is driving it to B (🟡 partially working / ✅ fully working)

| Feature | Subfeatures / details | Status | Notes (PR) |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Materializations: `distributed_table` / `distributed_incremental`** | *(overall)* | 🔴 | Macro bodies merged (dbt-labs/dbt-core#16079) but unusable: every distributed model aborts at the compile-time cluster guard because the ON CLUSTER machinery (#711) is missing. The incremental-strategy/`on_schema_change` pieces (dbt-labs/dbt-core#16090) and projections (dbt-labs/dbt-core#16200) exist but stay blocked behind it |
| `insert_distributed_sync` expectations | Compile-time guard + client-side connection setting | ✅ | Guard merged (queries `system.settings`, errors unless `insert_distributed_sync=1`); the per-connection setting is sent as a driver option (dbt-labs/dbt-core#16156), so no server-side override is needed |
| `adapter.get_clickhouse_local_suffix` / `get_clickhouse_local_db_prefix` | Local-table naming (profile `local_suffix`/`local_db_prefix`) | 🟡 | Profile keys parse (defaults `_local` / empty) and the free-macro getters exist and are used by both materializations. Missing: the v1 `adapter.*`-method form; unreachable anyway while the guard aborts first |
| Incremental strategies on `distributed_incremental` | `legacy` (default), `append`, `delete_insert`, `insert_overwrite` | 🔴 | Strategy resolution and the distributed hunks merged (dbt-labs/dbt-core#16090); blocked at the cluster guard. `legacy` additionally needs server-side `distributed_product_mode='local'` (or GLOBAL IN), as in v1 |
| `on_schema_change` on `distributed_incremental` | append_new_columns / sync_all_columns / fail | 🔴 | `check_incremental_schema_changes` + `clickhouse__apply_column_changes` on the local relation merged (dbt-labs/dbt-core#16090); blocked at the cluster guard |
| Base flow | Distributed wrapper + `_local` tables on every shard | 🔴 | Full macro flow (view tmp → local table create/exchange-or-rename → Distributed wrapper → insert) merged; blocked at the cluster guard |
| `sharding_key` config | Distributed engine sharding expression (defaults `rand()`) | 🔴 | Key unregistered → `dbt1060 Ignored unexpected key "sharding_key"` |
| `inserts_only` fast path | Plain INSERT into the Distributed wrapper | 🔴 | Branch present in the macro; blocked at the cluster guard |
| Projections in local-table DDL | `projections` config in `create_empty_table_from_relation` + up-front `validate_projections` | 🔴 | Main renders only the plain `PROJECTION name (query)` form. The shared `clickhouse_projection_ddl` renderer (incl. `index` form) and `validate_projections` are merged (dbt-labs/dbt-core#16200) for the plain table path; the distributed call sites still need the cluster machinery |
| `query_settings` on introspection | `get_column_schema_from_query(sql, query_settings=…)` in local-table DDL | 🔴 | The kwarg is merged for non-distributed paths (dbt-labs/dbt-core#16079); the distributed call site does not forward it yet |

Contributor guide

Open the contributing guide

Research direction

Start with the distributed materialization macros and the cluster machinery tracked in #711. Trace the compile-time cluster guard, then inspect the distributed config registration for `sharding_key` and the call sites for projections and `query_settings`. Done means distributed models can compile and run with a configured cluster, and the listed distributed options are accepted and applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.