ClickHouse / ClickHouse/dbt-clickhouse

Add ClickHouse support in dbt Core 2.0

Open
#660 0 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

Issue created to specifically track the development of the ClickHouse adapter in dbt Core 2.0. Mirror issue from dbt Core repository: [https://github.com/dbt-labs/dbt-core/issues/14585]()

# Context

**Small clarification on dbt Core 1.X vs dbt Core 2.X vs dbt Fusion**

* This repo contains the implementation for dbt Core 1.X (the Python version). This version will continue to be maintained and improved from this repository.
* dbt Core 2.0 is a full redesign of the dbt engine and lives alongside the adapter implementations in a different repository: [https://github.com/dbt-labs/dbt-core](). Once production-ready, upgrading will work just like before. Simply update (or build) your `dbt` binary and you can continue using your existing project!
* dbt Fusion: This is dbt Core 2.0 plus a set of interesting features, including more intelligent SQL parsing and a VS Code extension [https://docs.getdbt.com/docs/fusion/about-fusion](). Fusion support is being tracked here: [https://github.com/ClickHouse/dbt-clickhouse/issues/555]().

**Current status**
ClickHouse is already supported in dbt Core 2.0 and most things are working!

That said, both dbt Core 2.0 and the ClickHouse adapter for that version **are in alpha and not production-ready yet**. Please be careful when testing so it doesn't interfere with your production workflows or files such as `manifest.json`.

You can find more details about the (upcoming) upgrade guides here: [https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v2?version=2.0&name=Fusion]()

**What you can expect right now**

* Basic projects should work without issues, but expect some buggy behavior and some inconsistencies.
* Materialized views (MV) now have v1 feature parity (see the `materialized_view` sub-issue below).

We don't have firm dates yet, but we're aiming for a production-ready version of dbt Core 2.0 in the second half of 2026. We'll keep this issue updated. Feel free to ask any questions in the meantime.

**Related resources**
Issue depends on [https://github.com/ClickHouse/dbt-clickhouse/issues/555]().

# Known limitations

* **Reading `UUID` columns requires a recent ClickHouse server (26.7+).** dbt Core 2.0 receives query results in Arrow format, and older servers cannot convert `UUID` columns to Arrow output: any `SELECT` returning a `UUID` column fails with `Code: 50. DB::Exception: The type 'UUID' of a column '' is not supported for conversion into Arrow data format (UNKNOWN_TYPE)`. Use a recent ClickHouse version when testing dbt Core 2.0. This is a server-side limitation; there is no adapter-side workaround.
* **Self-managed clusters: `ON CLUSTER` DDL is not emitted yet.** What fails when the profile sets `cluster:`, and why ClickHouse Cloud is not affected, is in ClickHouse/dbt-clickhouse#711.

# Pending work for feature parity

Feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion, versus the Python `dbt-clickhouse` adapter. The work is split into **one sub-issue per area** — start here for the overall status, then rely on each sub-issue for the detailed feature tables and the actual pending changes. The sub-issue list on this issue shows live progress.

**Status legend:**

* ✅ implemented — merged and working on dbt-core 2.0 main
* 🟡 partial — partly working on main; see the sub-issue for what works vs fails
* 🚧 in progress — implemented in an open PR (linked in Notes)
* 🔴 not started — not working anywhere

| Area | Overall status | Sub-issue | Summary |
| ----------------------------------------------------------------- | -------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Materialization: `table` | ✅ | ClickHouse/dbt-clickhouse#702 | Everything table-specific works: v1 table skeleton, all DDL configs (`engine`/`order_by`/`ttl`/`settings`/`query_settings`/`primary_key`/`partition_by`), `indexes`/`projections` (incl. index-type), `sql_header`, EXCHANGE rebuilds, MV-target protection |
| Materialization: `view` | ✅ | ClickHouse/dbt-clickhouse#703 | Everything view-specific works (create/replace, `sql_security`/`definer`, auto-created-MV cleanup) |
| Materialization: `incremental` | ✅ | ClickHouse/dbt-clickhouse#704 | Merged: all strategies (`legacy`/`append`/`delete_insert`/`insert_overwrite`/`microbatch`), `unique_key`, `incremental_predicates`, Python-exact strategy resolution/validation, `on_schema_change` |
| Materialization: `materialized_view` | ✅ | ClickHouse/dbt-clickhouse#705 | Everything MV-specific works: create (incl. multiple MVs), MODIFY QUERY on re-run, `refreshable` clause + refresh-schedule updates, `depends_on` validation, `on_schema_change`, target protection (`mvs_pointing_to_it`), multi-MV update/rename, `refreshable: false`. The proper upstream multi-MV design is ⚪️ future work |
| Materialization: `dictionary` | ✅ | ClickHouse/dbt-clickhouse#706 | Done (byte-parity macro, all configs). Engine→relation-type mapping for existing dictionaries is ⚪️ future work |
| Materialization: `snapshot` | ✅ | ClickHouse/dbt-clickhouse#708 | Done for v1 parity (`timestamp`/`check` strategies, derivative upsert, tmp cleanup) |
| Materialization: `seed` | ✅ | ClickHouse/dbt-clickhouse#709 | Seed flow works end-to-end, incl. `'` and `?` in CSV data. Integer width inference and the `format CSV` insert path are ⚪️ future work |
| Materializations: `distributed_table` / `distributed_incremental` | 🔴 | ClickHouse/dbt-clickhouse#707 | Macros merged at near-v1 parity, but every distributed model aborts at the compile-time cluster guard: the ON CLUSTER machinery (cluster resolution, `should_on_cluster` — #711) and the `sharding_key` key are missing |
| Materialization: `clone` | 🔴 | ClickHouse/dbt-clickhouse#730 | Not on main (`clone.sql` absent; 5 tests fail) |
| Model contracts & constraints | ✅ | ClickHouse/dbt-clickhouse#729 | Complete on every runnable SQL materialization (dbt-labs/dbt-core#16082): DESCRIBE validation, `CONSTRAINT_SUPPORT` matrix, constraint DDL, TYPE_LABELS, column `codec`/`ttl`. Enforcement on distributed models waits on the cluster machinery (#707) |
| Tests / catalog / docs | 🟡 | ClickHouse/dbt-clickhouse#710 | Merged: data tests, test extra configs, catalog macros (`--write-catalog`), `persist_docs`, docs-generate suite, unit tests (dbt-labs/dbt-core#16153). Missing: `persist_docs` descriptions containing `;` (statement splitter), v1 `persist_docs.sql` re-sync |
| Cross-materialization settings | 🟡 | ClickHouse/dbt-clickhouse#711 | Merged: model `settings`/`query_settings`, version gating, dedup defaults, config-key registration for model keys, connection settings + `allow_nondeterministic_mutations` (dbt-labs/dbt-core#16156). Missing: ON CLUSTER (incl. `disable_on_cluster`), grants, source-table `database` key |
| General adapter behaviour | 🟡 | ClickHouse/dbt-clickhouse#712 | Merged: version probe, DESCRIBE column introspection, TLS, non-experimental onboarding, S3 (`clickhouse_s3source`), lw-delete probe, EXCHANGE probe (dbt-labs/dbt-core#16200), 0.1.1 driver with the literal-`?` fix (dbt-labs/dbt-core#16185), `dbt/` in the User-Agent (dbt-labs/dbt-core#16156; not always first). Missing: usage tracking (dbt first in the User-Agent), query comment, `query_id`, view replace fallback without `renameat2` |
| jaffle-shop validation | ✅ | ClickHouse/dbt-clickhouse#734 | Silent seed data loss (splitter dropped unlexable statements, so seed INSERTs with `'` in CSV data never ran) is fixed on main and the ClickHouse lexer routing + unit-test fix are merged (dbt-labs/dbt-core#16153); seed CSV values containing `?` load byte-exact since dbt-labs/dbt-core#16156: `seed`, `run` and `build` pass |
| dbt v2 specific functionality | 🔴 | ClickHouse/dbt-clickhouse#736 | Fusion-only capabilities: the SQL frontend has no ClickHouse dialect, so static analysis is forced off and the engine produces no column lineage. dbt platform column lineage works only with quoting disabled in `dbt_project.yml`, because its fallback parser rejects backtick-quoted identifiers. Metadata-based `dbt source freshness` is not implemented (panics) |
| Future work (explicitly deferred items) | ⚪️ | ClickHouse/dbt-clickhouse#731 | Collects every ⚪️-marked item, one table per area (seed integer width inference + `get_csv_data` insert path; dictionary engine→relation-type mapping; multiple-MVs upstream fix; 1.10.1/1.10.2 feature ports; engine-side CLI/config policies) |

## Tracking of issues from ADBC ClickHouse
All three fixed in adbc_clickhouse 0.1.1, on dbt Core 2.0 main via dbt-labs/dbt-core#16185:
- https://github.com/ClickHouse/adbc_clickhouse/issues/53
- https://github.com/ClickHouse/adbc_clickhouse/issues/67
- https://github.com/ClickHouse/adbc_clickhouse/issues/70

## Tracking of pending issues from clickhouse-rs
- https://github.com/ClickHouse/clickhouse-rs/issues/469 — `clickhouse-ext-arrow` puts its product info before dbt's in the User-Agent. Needs the fix, a clickhouse-rs release and an adbc_clickhouse release before dbt Core 2.0 can bump the driver pin

## Tracking of pending issues from ClickHouse
- https://github.com/ClickHouse/ClickHouse/issues/114130

# Extra steps to improve Fusion cohesion

- [ ] Improve how MVs are integrated in dbt
- [https://github.com/ClickHouse/dbt-clickhouse/issues/644]()
- [https://github.com/ClickHouse/dbt-clickhouse/issues/664]()
- [https://github.com/dbt-labs/dbt-core/issues/12319]()
- [ ] Move our integrations test suite to dbt Core 2.x
- [ ] Have the same tests as in `dbt-adapter-tests` for Core 2.0

Contributor guide

Open the contributing guide

Research direction

Start with the feature-parity table and choose a specific pending sub-issue, such as #707 for distributed materializations or #730 for clone support; the implementation lives with dbt Core 2.0 and this adapter. Read the selected sub-issue for its detailed feature table and tests; done means that area reaches the stated parity status.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.