ClickHouse / ClickHouse/dbt-clickhouse

Cannot change between view and distributed table on cluster

Open
#345 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug
We want to use dbt on a Clickhouse cluster with two nodes. As far as I can tell, you cannot switch from a view materialization to a distributed table, or vice versa.

### Steps to reproduce
profiles.yml is set up with on_cluster.

Model materialized as view:
``` sql
{{
config(
materialized='view',
)
}}

SELECT 1 as id
```
Run `dbt build`, successfully creates the view.
Switch model to distributed_table:
```sql
{{
config(
materialized='distributed_table',
order_by='id',
sharding_key='cityHash64(id)',
engine='ReplacingMergeTree'
)
}}

SELECT 1 as id
```
Run `dbt build` again, this fails:
```
Database Error in model test (models/test.sql)
:HTTPDriver for [hostname]:[port] returned response code 404)
Code: 60. DB::Exception: Table [schema].`test_local` doesn't exist. (UNKNOWN_TABLE) (version 24.1.8.22 (official build))
```
Also a run with --full-refresh does not help.

If you start with a clean distributed table (distributed + local table) and change the materialization to a view, it succeeds, by changing the test table to a view, but the test_local table remains.

### Expected behaviour

Changing materialization deletes and creates the expected tables/view.

### Configuration
#### Environment
* dbt version: 1.8.4
* dbt-clickhouse version: 1.8.1
* Python version: 3.11

#### ClickHouse server
* ClickHouse Server version: 24.1.8.22

Is this behaviour intended or expected? If not, a fix would be appreciated, as our current workaround for this is to drop and recreate the whole schema.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the view-to-distributed_table and distributed_table-to-view transitions with the shown profiles.yml and model configurations, using dbt build and --full-refresh. Trace the materialization behavior that handles the existing test and test_local tables. Done means switching materializations creates the expected object type and removes obsolete tables without requiring the whole schema to be dropped.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.