ClickHouse / ClickHouse/dbt-clickhouse

append_new_columns doesn't allow you to add new columns to a model

Open
#435 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug
`on_schema_change` is set to `append_new_columns` but when I add new columns, I get a NUMBER_OF_COLUMNS_DOESNT_MATCH error:

```
Database Error in model xxxxxxxxx (xxxxxxxxx)
--
  | HTTPDriver for xxxxxxxxx received ClickHouse error code 20
  | Code: 20. DB::Exception: Number of columns doesn't match (source: 15 and result: 14). (NUMBER_OF_COLUMNS_DOESNT_MATCH) (version 24.10.1.11366 (official build))
  | compiled code at xxxxxxxxxxx
```

I fixed this by going into Clickhouse directly and writing an `ALTER TABLE` query to add the new column. Then when I re materialised the asset, it worked

### Steps to reproduce
1. create an incremental model with `on_schema_change` = `append_new_columns`
2. materialise it
3. add a new column
4. materialise it again and you should see the above error

### Expected behaviour
When `on_schema_change` is set to `append_new_columns` and you add a new column to an already materialised model then it should allow you to add those columns

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the incremental model workflow described: materialise the model, add a column, and materialise it again with on_schema_change set to append_new_columns. Then trace the implementation of that schema-change path and verify that the existing table gains the new column without a NUMBER_OF_COLUMNS_DOESNT_MATCH error.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.