ClickHouse / ClickHouse/dbt-clickhouse

Cannot add new columns to incremental models

Open
#258 3 comments 3 reactions 0 assignees View on GitHub
bug good first issue
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug
When I try to add columns to a model in a distributed setup using MergeTree and ReplacingMergeTree, I am not able to add new columns.
Running the model for the first time succeeds. Running it again also works. Adding new columns and running it, produces the following error:

> Completed with 1 error and 0 warnings:
>
> Database Error in model test1 (models/test/test1.sql)
> :HTTPDriver for http://localhost:8123 returned response code 500)
> Code: 20. DB::Exception: Number of columns doesn't match. (NUMBER_OF_COLUMNS_DOESNT_MATCH) (version 24.1.5.6 (official build))

Related: #226

### Steps to reproduce
1. Run:
```
{{ config(engine='MergeTree()',
materialized='incremental',
on_schema_change='append_new_columns') }}

SELECT
1 AS column1,
1 AS column2
```
2. Then Run:
```
{{ config(engine='MergeTree()',
materialized='incremental',
on_schema_change='append_new_columns') }}

SELECT
1 AS column1,
1 AS column2,
1 AS column3
```

### Expected behaviour
Table in database should have had `column3` added

#### Environment
* dbt version: 1.7.8
* dbt-clickhouse version: 1.7.3
* clickhouse version: 24.1.5.6
* clickhouse-driver version (if using native): 0.2.7
* clickhouse-connect version (if using http): 0.7.0
* Python version: 3.12.2
* Operating system: MAC OS

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the two incremental model configurations and the listed dbt, dbt-clickhouse, and ClickHouse versions. Compare the first and second runs when column3 is introduced; done means the existing table gains column3 and the incremental run completes without NUMBER_OF_COLUMNS_DOESNT_MATCH.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.