ClickHouse / ClickHouse/dbt-clickhouse

dbt Core 2.0 parity: Cross-materialization settings

Open
#711 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
176
Avg merge
2d 10h
Merged PRs (30d)
8

Description

Sub-issue of #660 — feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion versus the Python `dbt-clickhouse` adapter.

Covers settings shared across materializations: the ON CLUSTER block, model/query settings, grants, config-key registration (a blocker for most other configs), connection settings and `allow_nondeterministic_mutations`.

**Status legend:** ✅ implemented — merged and working on dbt-core 2.0 `origin/main` · 🟡 partial — partly working on `origin/main`; Notes say what works vs what fails · 🚧 in progress — implemented in an open PR (linked in Notes) · 🔴 not started — not working anywhere · ⚪️ future work — explicitly deferred, not being solved now (also listed in the future-work sub-issue) · Combined status "A-🚧->B": the feature is at A on origin/main today; an open PR is driving it to B (🟡 partially working / ✅ fully working)

| Feature | Subfeatures / details | Status | Notes (PR) |
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cross-materialization settings** | *(overall)* | 🟡 | Merged: model `settings`/`query_settings`, version gating, dedup defaults (dbt-labs/dbt-core#16079), lw-deletes probe (dbt-labs/dbt-core#16090), model config-key registration, connection settings + `allow_nondeterministic_mutations` (dbt-labs/dbt-core#16156). Missing: ON CLUSTER (incl. the `disable_on_cluster` key), grants |
| `adapter.get_model_settings` | User `settings` + engine filter + dedup default `replicated_deduplication_window='0'` (unless `allow_automatic_deduplication`) | ✅ | dbt-labs/dbt-core#16079: renders the `-- end_of_sql` + `SETTINGS` block in CREATE TABLE DDL for table/incremental/ephemeral/materialized_view (explicit user value wins; MergeTree-only engine filter). Consumed by table.sql, view.sql (MV inner target), distributed_table.sql |
| `allow_automatic_deduplication` profile flag | Opts out of the `replicated_deduplication_window='0'` default | ✅ | dbt-labs/dbt-core#16079 |
| Server-version gating (`adapter.is_before_version` / `is_at_or_after_version`) | One `SELECT version()` probe per process; version-gated DDL | ✅ | dbt-labs/dbt-core#16079 (e.g. `insert_overwrite` emits `allow_replace_partition_from_empty_source=1` only on 26.6+) |
| `allow_nondeterministic_mutations` | Managed as in Python: probed, then sent when disabled-but-changeable | ✅ | Probe in dbt-labs/dbt-core#16090 (`probe_lightweight_deletes` mirrors dbclient.py; `use_lw_deletes` gates strategy resolution); sent as a `clickhouse.setting.*` statement option on every post-probe statement when disabled-but-changeable (dbt-labs/dbt-core#16156, on the 0.1.1 driver from dbt-labs/dbt-core#16185) |
| Connection settings (v1 dbclient `_conn_settings`) | `custom_settings` from profile; `mutations_sync='3'`, `alter_sync='3'`, `insert_distributed_sync='1'`, `lightweight_deletes_sync='3'`; `select_sequential_consistency` for Cloud/Shared; `insert_quorum`/`database_replicated_enforce_synchronous_settings` for Replicated/cluster_mode | ✅ | dbt-labs/dbt-core#16156 (driver options, plus the default database on the connection — v1 `_ensure_database` parity), on adbc_clickhouse 0.1.1 (#70 settings passthrough, #67 default database; dbt-labs/dbt-core#16185) |
| `query_settings` config and `adapter.get_model_query_settings` | Shared by view/table/incremental; `SETTINGS` clause on inserts and on contract introspection (`DESCRIBE TABLE () SETTINGS …`) | 🟡 | dbt-labs/dbt-core#16079. The lw-delete settings travel as connection settings, as in v1 (dbt-labs/dbt-core#16156). Missing: `query_settings` on seed inserts (v2's literal-VALUES path renders no SETTINGS clause; returns with the deferred `format CSV` insert path, #731) |
| Config-key registration | Fusion errors (dbt1060) and drops unknown model-config keys — every key must be registered in Rust before macros can see it | 🟡 | Registered: `engine`, `order_by`, `ttl`, `settings`, `query_settings`, `projections`, `inserts_only`, dictionary keys, `definer`, `sql_security`, `refreshable`, `catchup`, `mv_on_schema_change`, `repopulate_from_mvs_on_full_refresh`, column `codec` (dbt-labs/dbt-core#15500, #15506, #15507, #16023, #16079). Missing: `sharding_key` (distributed-only, #707) and `disable_on_cluster` (ON CLUSTER row below) — one such key aborts the whole run; a per-table `database:` on a source (`dbt1060 Ignored unexpected key "database"`, so a source cannot point at another ClickHouse database); project-level `+keys` under `seeds:` (dropped by `ProjectSeedConfig`); properties-level fixture keys (seed `quote_columns`, node `docs`/`meta`/`tags`) error and break docs generate (#710); `sharding_key`/`disable_on_cluster` on `DataTestConfig`/`UnitTestConfig`. Engine-policy divergences (model-yml `docs:`, exposure `meta:`/`tags:`, unprefixed dbt_project.yml keys) are deferred |
| ON CLUSTER support | Profile `cluster` + `database_engine` read at adapter construction; `on_cluster_clause` on DDL; per-model `disable_on_cluster` opt-out | 🔴 | Profile keys parse and `on_cluster_clause` exists in the macros, but the adapter cluster getters are stubs (cluster name → none, `should_on_cluster` → false) and `relation.should_on_cluster` does not exist, so `ON CLUSTER` never renders; the `disable_on_cluster` key is unregistered (`dbt1060`). **Who is affected: only self-managed multi-node deployments that set `cluster:` in the profile** — see "ON CLUSTER: what fails today" below. **ClickHouse Cloud is not affected**: v1 emits ON CLUSTER only when the profile sets `cluster:` (and never on a Replicated database), Cloud profiles set no cluster, so v1 and v2 behave identically there with no special-casing needed |
| `relation.can_on_cluster` / `should_on_cluster` | Stamped from database engine in catalog paths; identifier-less ⇒ true | 🔴 | Not on main (attribute undefined → falsy) |
| `adapter.get_clickhouse_cluster_name` / `clickhouse_db_engine_clause` | Cluster name must be double-quoted | 🔴 | Dispatch arms are explicit stubs (`get_clickhouse_cluster_name` → None, `clickhouse_db_engine_clause` → `""`, `should_on_cluster` → false) |
| Grants | `apply_grants.sql` (show/grant/revoke, ON CLUSTER) + ClickHouse arm in `standardize_grants_dict` + `toString(access_type)` (Enum16 arrives as Int16 via Arrow) | 🔴 | No ClickHouse `apply_grants.sql`, so the default Postgres-shaped `grant … on` SQL is a syntax error (Model/Seed/Incremental/Snapshot grants fail). The table.sql call site exists (dbt-labs/dbt-core#16082). Related: dbt-labs/dbt-core#14652 |
| Typed-config parse errors inside `config()` | A config value with the wrong shape (e.g. a bool where a map is expected) must fail the node | ⚪️ | Deferred (#731). The node succeeds with its whole `config()` block dropped instead of failing, so e.g. a `materialized_view` silently becomes a plain view |

#### ON CLUSTER: what fails today (self-managed clusters with `cluster:` in the profile)

- **Replicated engines on an Atomic database fail loudly**: any `Replicated*MergeTree` model, seed or snapshot DDL is rejected with `Macro 'uuid' in engine arguments is only supported … within an ON CLUSTER query`.
- **Distributed materializations cannot run**: `distributed_table` / `distributed_incremental` abort at their compile-time guard (`To use distributed materialization cluster setting in dbt profile must be set`) because the guard reads the stubbed cluster name, not the profile (#707).
- **Silent single-node divergence where DDL does succeed**: plain MergeTree tables, views, dictionaries, `create schema`, drop, rename and `on_schema_change` ALTERs run without ON CLUSTER, so the objects exist only on the node the connection hit. No error is raised — this is the dangerous case when migrating a working v1 project.
- **`disable_on_cluster` aborts the whole run**: the per-model opt-out key is unregistered, so a project that uses it fails at parse with `dbt1060`.
- **Replicated *database* engine (`database_engine: Replicated`) is mostly unaffected**: v1 also omits ON CLUSTER there because the database replicates DDL itself, and `create schema` on main does render `ENGINE = Replicated` from the profile; only the distributed guard and the `disable_on_cluster` key still bite.
- Not part of this gap: `insert_distributed_sync=1` already travels as a connection setting (dbt-labs/dbt-core#16156).

#### Related issues:
- Grants code needs `toString(access_type)` because ADBC/Arrow returns Enum values as their underlying integer: https://github.com/ClickHouse/ClickHouse/issues/114130

Contributor guide

Open the contributing guide

Research direction

Start with the missing ON CLUSTER and grants entries in this parity tracker, then inspect the referenced table.sql, view.sql, and distributed_table.sql templates and the existing dbt-core parity changes. Check the seed insert path for query_settings as well. Done means the remaining settings are registered and rendered consistently with the stated Python-adapter behavior, with coverage for the affected materializations.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.