ClickHouse / ClickHouse/dbt-clickhouse
Add ClickHouse support in dbt Core 2.0
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
Issue created to specifically track the development of the ClickHouse adapter in dbt Core 2.0. Mirror issue from dbt Core repository: [https://github.com/dbt-labs/dbt-core/issues/14585]()
# Context
**Small clarification on dbt Core 1.X vs dbt Core 2.X vs dbt Fusion**
* This repo contains the implementation for dbt Core 1.X (the Python version). This version will continue to be maintained and improved from this repository.
* dbt Core 2.0 is a full redesign of the dbt engine and lives alongside the adapter implementations in a different repository: [https://github.com/dbt-labs/dbt-core](). Once production-ready, upgrading will work just like before. Simply update (or build) your `dbt` binary and you can continue using your existing project!
* dbt Fusion: This is dbt Core 2.0 plus a set of interesting features, including more intelligent SQL parsing and a VS Code extension [https://docs.getdbt.com/docs/fusion/about-fusion](). Fusion support is being tracked here: [https://github.com/ClickHouse/dbt-clickhouse/issues/555]().
**Current status**
ClickHouse is already supported in dbt Core 2.0 and most things are working!
That said, both dbt Core 2.0 and the ClickHouse adapter for that version **are in alpha and not production-ready yet**. Please be careful when testing so it doesn't interfere with your production workflows or files such as `manifest.json`.
You can find more details about the (upcoming) upgrade guides here: [https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v2?version=2.0&name=Fusion]()
**What you can expect right now**
* Basic projects should work without issues, but expect some buggy behavior and some inconsistencies.
* Materialized views (MV) now have v1 feature parity (see the `materialized_view` sub-issue below).
We don't have firm dates yet, but we're aiming for a production-ready version of dbt Core 2.0 in the second half of 2026. We'll keep this issue updated. Feel free to ask any questions in the meantime.
**Related resources**
Issue depends on [https://github.com/ClickHouse/dbt-clickhouse/issues/555]().
# Known limitations
* **Reading `UUID` columns requires a recent ClickHouse server (26.7+).** dbt Core 2.0 receives query results in Arrow format, and older servers cannot convert `UUID` columns to Arrow output: any `SELECT` returning a `UUID` column fails with `Code: 50. DB::Exception: The type 'UUID' of a column '' is not supported for conversion into Arrow data format (UNKNOWN_TYPE)`. Use a recent ClickHouse version when testing dbt Core 2.0. This is a server-side limitation; there is no adapter-side workaround.
* **Self-managed clusters: `ON CLUSTER` DDL is not emitted yet.** What fails when the profile sets `cluster:`, and why ClickHouse Cloud is not affected, is in ClickHouse/dbt-clickhouse#711.
# Pending work for feature parity
Feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion, versus the Python `dbt-clickhouse` adapter. The work is split into **one sub-issue per area** — start here for the overall status, then rely on each sub-issue for the detailed feature tables and the actual pending changes. The sub-issue list on this issue shows live progress.
**Status legend:**
* ✅ implemented — merged and working on dbt-core 2.0 main
* 🟡 partial — partly working on main; see the sub-issue for what works vs fails
* 🚧 in progress — implemented in an open PR (linked in Notes)
* 🔴 not started — not working anywhere
| Area | Overall status | Sub-issue | Summary |
| ----------------------------------------------------------------- | -------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Materialization: `table` | ✅ | ClickHouse/dbt-clickhouse#702 | Everything table-specific works: v1 table skeleton, all DDL configs (`engine`/`order_by`/`ttl`/`settings`/`query_settings`/`primary_key`/`partition_by`), `indexes`/`projections` (incl. index-type), `sql_header`, EXCHANGE rebuilds, MV-target protection |
| Materialization: `view` | ✅ | ClickHouse/dbt-clickhouse#703 | Everything view-specific works (create/replace, `sql_security`/`definer`, auto-created-MV cleanup) |
| Materialization: `incremental` | ✅ | ClickHouse/dbt-clickhouse#704 | Merged: all strategies (`legacy`/`append`/`delete_insert`/`insert_overwrite`/`microbatch`), `unique_key`, `incremental_predicates`, Python-exact strategy resolution/validation, `on_schema_change` |
| Materialization: `materialized_view` | ✅ | ClickHouse/dbt-clickhouse#705 | Everything MV-specific works: create (incl. multiple MVs), MODIFY QUERY on re-run, `refreshable` clause + refresh-schedule updates, `depends_on` validation, `on_schema_change`, target protection (`mvs_pointing_to_it`), multi-MV update/rename, `refreshable: false`. The proper upstream multi-MV design is ⚪️ future work |
| Materialization: `dictionary` | ✅ | ClickHouse/dbt-clickhouse#706 | Done (byte-parity macro, all configs). Engine→relation-type mapping for existing dictionaries is ⚪️ future work |
| Materialization: `snapshot` | ✅ | ClickHouse/dbt-clickhouse#708 | Done for v1 parity (`timestamp`/`check` strategies, derivative upsert, tmp cleanup) |
| Materialization: `seed` | ✅ | ClickHouse/dbt-clickhouse#709 | Seed flow works end-to-end, incl. `'` and `?` in CSV data. Integer width inference and the `format CSV` insert path are ⚪️ future work |
| Materializations: `distributed_table` / `distributed_incremental` | 🔴 | ClickHouse/dbt-clickhouse#707 | Macros merged at near-v1 parity, but every distributed model aborts at the compile-time cluster guard: the ON CLUSTER machinery (cluster resolution, `should_on_cluster` — #711) and the `sharding_key` key are missing |
| Materialization: `clone` | 🔴 | ClickHouse/dbt-clickhouse#730 | Not on main (`clone.sql` absent; 5 tests fail) |
| Model contracts & constraints | ✅ | ClickHouse/dbt-clickhouse#729 | Complete on every runnable SQL materialization (dbt-labs/dbt-core#16082): DESCRIBE validation, `CONSTRAINT_SUPPORT` matrix, constraint DDL, TYPE_LABELS, column `codec`/`ttl`. Enforcement on distributed models waits on the cluster machinery (#707) |
| Tests / catalog / docs | 🟡 | ClickHouse/dbt-clickhouse#710 | Merged: data tests, test extra configs, catalog macros (`--write-catalog`), `persist_docs`, docs-generate suite, unit tests (dbt-labs/dbt-core#16153). Missing: `persist_docs` descriptions containing `;` (statement splitter), v1 `persist_docs.sql` re-sync |
| Cross-materialization settings | 🟡 | ClickHouse/dbt-clickhouse#711 | Merged: model `settings`/`query_settings`, version gating, dedup defaults, config-key registration for model keys, connection settings + `allow_nondeterministic_mutations` (dbt-labs/dbt-core#16156). Missing: ON CLUSTER (incl. `disable_on_cluster`), grants, source-table `database` key |
| General adapter behaviour | 🟡 | ClickHouse/dbt-clickhouse#712 | Merged: version probe, DESCRIBE column introspection, TLS, non-experimental onboarding, S3 (`clickhouse_s3source`), lw-delete probe, EXCHANGE probe (dbt-labs/dbt-core#16200), 0.1.1 driver with the literal-`?` fix (dbt-labs/dbt-core#16185), `dbt/` in the User-Agent (dbt-labs/dbt-core#16156; not always first). Missing: usage tracking (dbt first in the User-Agent), query comment, `query_id`, view replace fallback without `renameat2` |
| jaffle-shop validation | ✅ | ClickHouse/dbt-clickhouse#734 | Silent seed data loss (splitter dropped unlexable statements, so seed INSERTs with `'` in CSV data never ran) is fixed on main and the ClickHouse lexer routing + unit-test fix are merged (dbt-labs/dbt-core#16153); seed CSV values containing `?` load byte-exact since dbt-labs/dbt-core#16156: `seed`, `run` and `build` pass |
| dbt v2 specific functionality | 🔴 | ClickHouse/dbt-clickhouse#736 | Fusion-only capabilities: the SQL frontend has no ClickHouse dialect, so static analysis is forced off and the engine produces no column lineage. dbt platform column lineage works only with quoting disabled in `dbt_project.yml`, because its fallback parser rejects backtick-quoted identifiers. Metadata-based `dbt source freshness` is not implemented (panics) |
| Future work (explicitly deferred items) | ⚪️ | ClickHouse/dbt-clickhouse#731 | Collects every ⚪️-marked item, one table per area (seed integer width inference + `get_csv_data` insert path; dictionary engine→relation-type mapping; multiple-MVs upstream fix; 1.10.1/1.10.2 feature ports; engine-side CLI/config policies) |
## Tracking of issues from ADBC ClickHouse
All three fixed in adbc_clickhouse 0.1.1, on dbt Core 2.0 main via dbt-labs/dbt-core#16185:
- https://github.com/ClickHouse/adbc_clickhouse/issues/53
- https://github.com/ClickHouse/adbc_clickhouse/issues/67
- https://github.com/ClickHouse/adbc_clickhouse/issues/70
## Tracking of pending issues from clickhouse-rs
- https://github.com/ClickHouse/clickhouse-rs/issues/469 — `clickhouse-ext-arrow` puts its product info before dbt's in the User-Agent. Needs the fix, a clickhouse-rs release and an adbc_clickhouse release before dbt Core 2.0 can bump the driver pin
## Tracking of pending issues from ClickHouse
- https://github.com/ClickHouse/ClickHouse/issues/114130
# Extra steps to improve Fusion cohesion
- [ ] Improve how MVs are integrated in dbt
- [https://github.com/ClickHouse/dbt-clickhouse/issues/644]()
- [https://github.com/ClickHouse/dbt-clickhouse/issues/664]()
- [https://github.com/dbt-labs/dbt-core/issues/12319]()
- [ ] Move our integrations test suite to dbt Core 2.x
- [ ] Have the same tests as in `dbt-adapter-tests` for Core 2.0
Contributor guide
Research direction
Start with the feature-parity table and choose a specific pending sub-issue, such as #707 for distributed materializations or #730 for clone support; the implementation lives with dbt Core 2.0 and this adapter. Read the selected sub-issue for its detailed feature table and tests; done means that area reaches the stated parity status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100