ClickHouse / ClickHouse/dbt-clickhouse

dbt Core 2.0: dbt v2 specific functionality

Open
#736 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

Sub-issue of #660. Functionality that only exists in dbt Core 2.0 / Fusion / dbt platform (no dbt Core 1.x counterpart in `dbt-clickhouse`), tracked here so the ClickHouse adapter can support it as well. Unlike the parity sub-issues, most of these items depend on engine or platform pieces owned by dbt Labs; the Notes column says which part is on our side and which is not.

**Status legend:**

- ✅ implemented — merged and working on dbt-core 2.0 origin/main
- 🟡 partial — partly working on origin/main; Notes say what works vs fails
- 🚧 in progress — implemented in an open PR (linked in Notes)
- 🔴 not started — not working anywhere

| Feature | What it means | Status | Notes |
|---|---|---|---|
| **dbt v2 specific functionality** | *(overall)* | 🔴 | Everything built on the engine's SQL understanding is off for ClickHouse because the SQL frontend has no ClickHouse dialect; only dbt platform column lineage works, and only with a project-level quoting workaround. Metadata-based source freshness is not implemented (`dbt source freshness` panics) |
| Column-level lineage in dbt platform | dbt platform's column lineage view, built from the artifacts a dbt Core 2.0 run produces | 🟡 | **Works** when the root `dbt_project.yml` disables quoting (`quoting: {database: false, schema: false, identifier: false}`), so compiled SQL renders relations as `default.my_model`. **Fails** with the default ClickHouse quoting: the lineage inspection reports `SQL parsing error — Invalid expression / Unexpected token` at the first backtick-quoted relation in `compiled_code` (e.g. `select * from \`default\`.\`my_first_dbt_model\``). Because the engine writes no column-lineage artifacts for ClickHouse (see the engine-side row), dbt platform falls back to parsing `compiled_code` with a dialect that rejects backtick identifiers. Fix is on the dbt Labs side: a ClickHouse dialect mapping in the platform parser. The quoting workaround is diagnostic only: unquoted identifiers break relation names that need quoting in ClickHouse |
| ClickHouse dialect in the SQL frontend | The engine's dialect-aware SQL layer (`Dialect` enum, `dialect_of(adapter_type)`, per-dialect ANTLR lexer) that every static feature below is built on | 🔴 | Missing: (1) an ANTLR lexer crate `dbt-lexer-clickhouse` under `crates/dbt-sql/` (exposing the `SEMI_COLON`/`UNPAIRED_TOKEN` tokens the statement splitter dispatches on, registered in `dbt-sql-utils`' `SUPPORTED_DIALECTS` and lexer dispatch); (2) a `Dialect::ClickHouse` variant plus the `dialect_of(ClickHouse)` arm (`crates/dbt-common/src/adapter.rs`, today `None`). The lexer is buildable from open inputs: the Rust-target ANTLR tool is released at sdf-labs/antlr4, the `dbt-antlr4` runtime is on crates.io, and ClickHouse's own `utils/antlr/ClickHouseLexer.g4` is the grammar (needs an `UNPAIRED_TOKEN` catch-all and a `$tag$` heredoc rule added). The other lexers' grammars live in closed parser crates and the semantic-analysis frontend that consumes `Dialect` is closed source, so the variant and the grammar's placement need dbt Labs. Unblocks: correct splitting of ClickHouse SQL, including `$tag$` heredocs (`persist_docs`, #710) — dbt-labs/dbt-core#16153 maps ClickHouse to the Databricks lexer as an interim for backslash escapes only — and the `dialect_of`-gated engine paths that return early for ClickHouse today (state-aware defer / run-cache relation parsing, SQL source extraction) |
| Static analysis (`--static-analysis baseline` / `strict`) | Compile-time SQL validation, schema inference and type checking without hitting the warehouse; drives the metadata index (`--write-metadata`), unit-test and data-test static evaluation | 🔴 | Forced to `off` for every ClickHouse invocation: the adapter is not in `STATIC_ANALYSIS_SUPPORTED_ADAPTERS` (`crates/dbt-adapter-core/src/lib.rs`) and `disable_static_analysis_if_not_supported` overrides any `--static-analysis` flag or `static_analysis` model config. Visible in artifacts as `static_analysis_off_reason: "configuredoff"` on every node. Depends on the dialect row |
| Column-level lineage (engine-side, `dbt docs` / parquet) | `--write-lineage` column lineage parquet consumed by the local docs site (`dbt docs generate`) and by dbt platform when present | 🔴 | The `column_lineage` hook returns no edges when static analysis is off, and the run logs `column-level lineage requires --static-analysis strict; no column lineage written` (`crates/dbt-main/src/dbt_lib.rs`). The generated docs site therefore has no column lineage for ClickHouse projects. Depends on the static analysis row |
| Source freshness from warehouse metadata | `dbt source freshness` on sources without `loaded_at_field`: dbt Core 2.0 asks the adapter for each source table's last modification time (`freshness_inner` → `last_altered`/`is_view`) instead of running a query | 🔴 | `ClickHouseMetadataAdapter::freshness_inner` is `todo!()`, so `dbt source freshness` panics for ClickHouse sources (dbt-labs/dbt-core#14653). Ours to implement: read `metadata_modification_time` from `system.tables` (or the max part `modification_time` from `system.parts`) for the requested relations. dbt Core 1.x never had a metadata path for ClickHouse (only the `loaded_at_field` query), so this is new functionality, not parity |

Upstream: dbt-labs/dbt-core#14653 (source freshness).

Contributor guide

Open the contributing guide

Research direction

Start with ClickHouseMetadataAdapter::freshness_inner and the source-freshness path described in the issue; inspect how dbt Core 2.0 requests last modification times and how ClickHouse system metadata exposes them. Run dbt source freshness against sources without loaded_at_field, then verify it no longer panics and reports freshness. The dialect, static-analysis, and lineage rows depend on dbt Labs-owned engine or platform changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.