ClickHouse / ClickHouse/dbt-clickhouse

dbt Core 2.0 parity: future work (explicitly deferred items)

Open
#731 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
176
Avg merge
2d 10h
Merged PRs (30d)
8

Description

Sub-issue of #660. Collects every feature explicitly deferred — decided NOT to be solved now (⚪️ in its home sub-issue). Each item appears twice on purpose: in its home sub-issue's table (marked ⚪️, as a one-line summary of the problem) and here, grouped by area and with the context (root cause, upstream links, workarounds), so the deferred backlog is readable in one place.

One table per area; areas with no deferred items yet say *(none yet)*.

## Materialization: `table` (#702)

*(none yet)*

## Materialization: `view` (#703)

*(none yet)*

## Materialization: `incremental` (#704)

*(none yet)*

## Materialization: `materialized_view` (#705)

| Feature | What was deferred | Notes |
|---|---|---|
| Multiple MVs per model: proper upstream design | Replacing the `--:begin/end` marker convention with a first-class multi-MV model design | Fresh create and the update/rename flows work with the current convention (dbt-labs/dbt-core#15507, #16200); the redesign depends on upstream dbt (dbt-labs/dbt-core#12319); moving the MV control-plane comment markers (`-- :begin/end`, `-- materialization_target_table:`) into `config()` is tracked in ClickHouse/dbt-clickhouse#644 and dbt-labs/dbt-core#14587 |

## Materialization: `dictionary` (#706)

| Feature | What was deferred | Notes |
|---|---|---|
| Engine → relation-type mapping for existing dictionaries | `relation_type_from_engine` mapping `system.tables` engine `Dictionary` → `RelationType::Dictionary` | `list_relations` maps every `system.tables` engine it does not know to `table`, so an existing dictionary comes back typed `table`. Catalog typing and any macro path branching on `relation.type == 'dictionary'` see the wrong type. Plain re-runs are unaffected because `CREATE OR REPLACE DICTIONARY` does not consult the cached type. Needs `RelationType::Dictionary` (already on main) wired into the engine mapping |

## Materializations: `distributed_table` / `distributed_incremental` (#707)

*(none yet)*

## Materialization: `snapshot` (#708)

*(none yet)*

## Materialization: `seed` (#709)

| Feature | What was deferred | Notes |
|---|---|---|
| Unused `adapter.get_csv_data` / v1 `format CSV` insert path | Restoring v1's insert path (currently batched literal-VALUES) | The literal-`VALUES` path exists because the 0.1.0 driver treated a literal `?` in SQL as a bind parameter (adbc_clickhouse#53), which made v1's raw-CSV insert unusable. That driver fix is on main (dbt-labs/dbt-core#16185) and the seed `??` escape is gone (dbt-labs/dbt-core#16156), but the restoration was not part of either: `clickhouse__load_csv_rows` must change on both the v1 macro tree and the v2 copy together to keep byte-identity |
| Integer CSV column width inference: `Int64` vs Python's `Int32` | Arrow-based inference types integer/float seed columns as `Int64`/`Float64` where v1's agate gives `Int32`/`Float32` | Nullability was fixed in dbt-labs/dbt-core#15912 (seed columns are non-`Nullable` like v1); only the width differs. Workaround: pin exact types with `column_types`. Revisit only if the width difference bites a real user |

## Tests / catalog / docs (#710)

| Feature | What was deferred | Notes |
|---|---|---|
| `dbt docs generate` command parity | `docs generate --no-compile` and docs-site asset copying (`asset-paths` → `target/assets`) | v2 has no `docs generate` command; the test shim maps it to `compile --write-catalog`, which produces the catalog but knows no `--no-compile` and copies no assets. The catalog content itself matches dbt-core's expected output once the fixtures use 1.10 yml (`docs`/`meta`/`tags` under `config`), so what is missing is the command surface, an engine CLI decision |
| Unit test `given` on a seed-backed `source()` read before the seed has run | Ordering the unit test's render-time schema fetch after the seed that creates the `given` relation (static analysis off) | With static analysis off (always the case for ClickHouse) a unit test fetches its `given` schemas from the database while rendering; when the `given` is a `source()` created by a seed of the same `dbt build`, the task graph does not order that fetch after the seed. Engine task-graph bug reported as dbt-labs/dbt-core#16202; waits for the core team. Shows up only when DDL is slow (seen on ClickHouse Cloud with jaffle-shop, #734) |
| Unit-test schema inference without adapter special cases | Making the engine's unit-test renderable generic where dbt-labs/dbt-core#16153 hardcodes `AdapterType::ClickHouse` | Two special cases in `crates/dbt-tasks-sa/src/renderable/renderable/unit_test.rs`: (1) the expected-schema probe creates a session-scoped TEMPORARY table and introspects it, which ClickHouse never lists in `system.columns`, so ClickHouse is forced onto the query-schema path DuckDB already uses — the generic fix is to infer from the query schema for every adapter or gate the probe on an adapter capability; (2) the persisted schema cache has no TTL and no invalidation, so a `given`/expected schema goes stale when an upstream model is rebuilt with different column types between invocations, and ClickHouse forces a refetch of every fixture schema. The forced refetch is the part to roll back: per the review thread (https://github.com/dbt-labs/dbt-core/pull/16153#discussion_r3950758691) it effectively disables the schema cache for ClickHouse to work around engine schema-caching bugs, so it should go once the cache invalidates on rebuild. Engine code, for the core team |
| `compile --write-catalog` panics in debug builds | Restoring the invocation span in the catalog worker threads so their query spans have a root | Regression from the engine change that stopped abandoned catalog workers from swallowing the run summary: the workers no longer enter the invocation span, so the adapter's per-query debug span becomes a parentless root and a `debug_assert!` in `dbt-tracing` aborts the process right after `Fetching catalog from warehouse`. Release binaries do not panic but emit those spans as orphaned trace roots. Generic ADBC engine path, not ClickHouse code; reported as dbt-labs/dbt-oss#16223 and left to the core team |
## Cross-materialization settings (#711)

| Feature | What was deferred | Notes |
|---|---|---|
| Typed-config parse errors inside `config()` must fail the node | A wrong-shaped config value (e.g. a bool where a map is expected) is logged as dbt1501 but the node renders as success with its whole `config()` block dropped | v2 logs the parse failure as dbt1501 and continues: the node renders as success with no config at all, so a `materialized_view` model silently materializes as a plain view. dbt-core fails the node. Engine config-parsing behaviour (`dbt-schemas` typed configs), not adapter code, so the fix belongs to the core team |

## General adapter behaviour (#712)

| Feature | What was deferred | Notes |
|---|---|---|
| Port dbt-clickhouse 1.10.1 features | Bring the features added in v1 1.10.1 to 2.X | Not itemized yet — needs a diff of the 1.10.1 changelog against the v2 feature tables |
| Port dbt-clickhouse 1.10.2 features | Bring the features added in v1 1.10.2 to 2.X | Not itemized yet — same exercise as 1.10.1 |
| `dbt debug` unknown-target report | Surfacing a missing/misconfigured `--target` as `dbt debug` check lines like dbt-core | v2 resolves the profile before `dbt debug` starts its report, so an unknown `--target` fails profile loading (`dbt1005`) and the connection-check lines dbt-core would print never appear. Engine CLI behaviour, not adapter code |
| `dbt run --sample` (sample mode) | Filtering models' inputs by the `--sample` window | The sampled model returns 0 rows where dbt-core returns the rows inside the `--sample` window (`TestSampleMode`, `model_that_samples_input_sql` comes back empty). The engine's sample-mode rewriting is not wired for ClickHouse; needs investigation on the engine side before any adapter work |
| Duplicate hook definitions | Failing compilation when a node defines both `post-hook` and `post_hook` (or the `pre-` pair) | dbt-core raises a compilation error; v2 accepts the duplicate silently and keeps one (`TestDuplicateHooksInConfigs`). Engine config-parsing policy: both spellings map to the same `ModelConfig` field (serde alias) and nothing diagnoses the pair |

Contributor guide

Open the contributing guide

Research direction

This issue is a deferred backlog rather than one bounded change. Select one table row first, then read its named context and entry point, such as clickhouse__load_csv_rows or crates/dbt-tasks-sa/src/renderable/renderable/unit_test.rs; confirm scope with the linked upstream issue where one exists. Done means the selected parity gap is implemented and its affected behavior is covered by the named test or command.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
databases, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.