ClickHouse / ClickHouse/dbt-clickhouse
Cannot add new columns to incremental models
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
When I try to add columns to a model in a distributed setup using MergeTree and ReplacingMergeTree, I am not able to add new columns.
Running the model for the first time succeeds. Running it again also works. Adding new columns and running it, produces the following error:
> Completed with 1 error and 0 warnings:
>
> Database Error in model test1 (models/test/test1.sql)
> :HTTPDriver for http://localhost:8123 returned response code 500)
> Code: 20. DB::Exception: Number of columns doesn't match. (NUMBER_OF_COLUMNS_DOESNT_MATCH) (version 24.1.5.6 (official build))
Related: #226
### Steps to reproduce
1. Run:
```
{{ config(engine='MergeTree()',
materialized='incremental',
on_schema_change='append_new_columns') }}
SELECT
1 AS column1,
1 AS column2
```
2. Then Run:
```
{{ config(engine='MergeTree()',
materialized='incremental',
on_schema_change='append_new_columns') }}
SELECT
1 AS column1,
1 AS column2,
1 AS column3
```
### Expected behaviour
Table in database should have had `column3` added
#### Environment
* dbt version: 1.7.8
* dbt-clickhouse version: 1.7.3
* clickhouse version: 24.1.5.6
* clickhouse-driver version (if using native): 0.2.7
* clickhouse-connect version (if using http): 0.7.0
* Python version: 3.12.2
* Operating system: MAC OS
Contributor guide
Research direction
Start by reproducing the issue with the two incremental model configurations and the listed dbt, dbt-clickhouse, and ClickHouse versions. Compare the first and second runs when column3 is introduced; done means the existing table gains column3 and the incremental run completes without NUMBER_OF_COLUMNS_DOESNT_MATCH.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100