ClickHouse / ClickHouse/dbt-clickhouse
append_new_columns doesn't allow you to add new columns to a model
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
`on_schema_change` is set to `append_new_columns` but when I add new columns, I get a NUMBER_OF_COLUMNS_DOESNT_MATCH error:
```
Database Error in model xxxxxxxxx (xxxxxxxxx)
--
| HTTPDriver for xxxxxxxxx received ClickHouse error code 20
| Code: 20. DB::Exception: Number of columns doesn't match (source: 15 and result: 14). (NUMBER_OF_COLUMNS_DOESNT_MATCH) (version 24.10.1.11366 (official build))
| compiled code at xxxxxxxxxxx
```
I fixed this by going into Clickhouse directly and writing an `ALTER TABLE` query to add the new column. Then when I re materialised the asset, it worked
### Steps to reproduce
1. create an incremental model with `on_schema_change` = `append_new_columns`
2. materialise it
3. add a new column
4. materialise it again and you should see the above error
### Expected behaviour
When `on_schema_change` is set to `append_new_columns` and you add a new column to an already materialised model then it should allow you to add those columns
Contributor guide
Research direction
Start by reproducing the incremental model workflow described: materialise the model, add a column, and materialise it again with on_schema_change set to append_new_columns. Then trace the implementation of that schema-change path and verify that the existing table gains the new column without a NUMBER_OF_COLUMNS_DOESNT_MATCH error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100