ClickHouse / ClickHouse/dbt-clickhouse
Cannot change between view and distributed table on cluster
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
We want to use dbt on a Clickhouse cluster with two nodes. As far as I can tell, you cannot switch from a view materialization to a distributed table, or vice versa.
### Steps to reproduce
profiles.yml is set up with on_cluster.
Model materialized as view:
``` sql
{{
config(
materialized='view',
)
}}
SELECT 1 as id
```
Run `dbt build`, successfully creates the view.
Switch model to distributed_table:
```sql
{{
config(
materialized='distributed_table',
order_by='id',
sharding_key='cityHash64(id)',
engine='ReplacingMergeTree'
)
}}
SELECT 1 as id
```
Run `dbt build` again, this fails:
```
Database Error in model test (models/test.sql)
:HTTPDriver for [hostname]:[port] returned response code 404)
Code: 60. DB::Exception: Table [schema].`test_local` doesn't exist. (UNKNOWN_TABLE) (version 24.1.8.22 (official build))
```
Also a run with --full-refresh does not help.
If you start with a clean distributed table (distributed + local table) and change the materialization to a view, it succeeds, by changing the test table to a view, but the test_local table remains.
### Expected behaviour
Changing materialization deletes and creates the expected tables/view.
### Configuration
#### Environment
* dbt version: 1.8.4
* dbt-clickhouse version: 1.8.1
* Python version: 3.11
#### ClickHouse server
* ClickHouse Server version: 24.1.8.22
Is this behaviour intended or expected? If not, a fix would be appreciated, as our current workaround for this is to drop and recreate the whole schema.
Contributor guide
Research direction
Start by reproducing the view-to-distributed_table and distributed_table-to-view transitions with the shown profiles.yml and model configurations, using dbt build and --full-refresh. Trace the materialization behavior that handles the existing test and test_local tables. Done means switching materializations creates the expected object type and removes obsolete tables without requiring the whole schema to be dropped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100