ClickHouse / ClickHouse/dbt-clickhouse
dbt Core 2.0 parity: Materializations `distributed_table` / `distributed_incremental`
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
Sub-issue of #660 — feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion versus the Python `dbt-clickhouse` adapter.
Covers the `distributed_table` and `distributed_incremental` materializations. Both macros are merged at near-v1 parity, **but no distributed model can run**: cluster resolution never succeeds on main, so both materializations abort at compile time with `To use distributed materialization cluster setting in dbt profile must be set`, even when the profile sets `cluster:`. The cluster machinery they depend on is tracked in #711; the distributed-only `sharding_key` config key is also missing. Both materializations are self-managed-cluster features; ClickHouse Cloud users are not affected by this gap.
**Status legend:** ✅ implemented — merged and working on dbt Core 2.0 `origin/main` · 🟡 partial — partly working on `origin/main` (Notes say what works vs. what fails) · 🚧 in progress — implemented in an open PR (linked in Notes) · 🔴 not started — not working anywhere · Combined status "A-🚧->B": the feature is at A on origin/main today; an open PR is driving it to B (🟡 partially working / ✅ fully working)
| Feature | Subfeatures / details | Status | Notes (PR) |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Materializations: `distributed_table` / `distributed_incremental`** | *(overall)* | 🔴 | Macro bodies merged (dbt-labs/dbt-core#16079) but unusable: every distributed model aborts at the compile-time cluster guard because the ON CLUSTER machinery (#711) is missing. The incremental-strategy/`on_schema_change` pieces (dbt-labs/dbt-core#16090) and projections (dbt-labs/dbt-core#16200) exist but stay blocked behind it |
| `insert_distributed_sync` expectations | Compile-time guard + client-side connection setting | ✅ | Guard merged (queries `system.settings`, errors unless `insert_distributed_sync=1`); the per-connection setting is sent as a driver option (dbt-labs/dbt-core#16156), so no server-side override is needed |
| `adapter.get_clickhouse_local_suffix` / `get_clickhouse_local_db_prefix` | Local-table naming (profile `local_suffix`/`local_db_prefix`) | 🟡 | Profile keys parse (defaults `_local` / empty) and the free-macro getters exist and are used by both materializations. Missing: the v1 `adapter.*`-method form; unreachable anyway while the guard aborts first |
| Incremental strategies on `distributed_incremental` | `legacy` (default), `append`, `delete_insert`, `insert_overwrite` | 🔴 | Strategy resolution and the distributed hunks merged (dbt-labs/dbt-core#16090); blocked at the cluster guard. `legacy` additionally needs server-side `distributed_product_mode='local'` (or GLOBAL IN), as in v1 |
| `on_schema_change` on `distributed_incremental` | append_new_columns / sync_all_columns / fail | 🔴 | `check_incremental_schema_changes` + `clickhouse__apply_column_changes` on the local relation merged (dbt-labs/dbt-core#16090); blocked at the cluster guard |
| Base flow | Distributed wrapper + `_local` tables on every shard | 🔴 | Full macro flow (view tmp → local table create/exchange-or-rename → Distributed wrapper → insert) merged; blocked at the cluster guard |
| `sharding_key` config | Distributed engine sharding expression (defaults `rand()`) | 🔴 | Key unregistered → `dbt1060 Ignored unexpected key "sharding_key"` |
| `inserts_only` fast path | Plain INSERT into the Distributed wrapper | 🔴 | Branch present in the macro; blocked at the cluster guard |
| Projections in local-table DDL | `projections` config in `create_empty_table_from_relation` + up-front `validate_projections` | 🔴 | Main renders only the plain `PROJECTION name (query)` form. The shared `clickhouse_projection_ddl` renderer (incl. `index` form) and `validate_projections` are merged (dbt-labs/dbt-core#16200) for the plain table path; the distributed call sites still need the cluster machinery |
| `query_settings` on introspection | `get_column_schema_from_query(sql, query_settings=…)` in local-table DDL | 🔴 | The kwarg is merged for non-distributed paths (dbt-labs/dbt-core#16079); the distributed call site does not forward it yet |
Contributor guide
Research direction
Start with the distributed materialization macros and the cluster machinery tracked in #711. Trace the compile-time cluster guard, then inspect the distributed config registration for `sharding_key` and the call sites for projections and `query_settings`. Done means distributed models can compile and run with a configured cluster, and the listed distributed options are accepted and applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100