ClickHouse / ClickHouse/dbt-clickhouse

`replicated_deduplication_window=0` is injected into manifest, breaking deferred runs

Open Beginner friendly
#676 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug

`get_model_settings` passes the `settings` dict from `model['config']` directly into `update_model_settings`, which mutates it in-place by injecting `replicated_deduplication_window='0'`. Because dbt serialises `manifest.json` after `dbt run`, this injected key gets persisted into the manifest.

The consequence is that commands like `dbt parse` or `dbt ls` — which never call `get_model_settings` — produce a clean manifest without the injected key. When using `--select state:modified+ --defer --state `, dbt compares the two manifests and sees `config_changed` on every affected model, even when nothing actually changed.

Related to #372 — that issue covers the unwanted DDL injection; this reports the deeper consequence of the in-place mutation leaking into the manifest.

### Steps to reproduce

```python
{{ config(
materialized='table',
engine='MergeTree()',
order_by='id',
settings={'allow_nullable_key': 1}
) }}
select 1 as id
```

1. `dbt run --select my_model` — produces `target/manifest.json` with `replicated_deduplication_window='0'` inside `config.settings`
2. `mkdir state && cp target/manifest.json state/`
3. `dbt run --select state:modified+ --defer --state state/` with no code changes

### Expected behaviour

Step 3 selects zero models — nothing changed. Actual: `my_model` is selected and re-run because dbt sees `config_changed` (the injected key is present in the state manifest but absent from the freshly parsed one).

### Code examples, such as models or profile settings

The diff between the manifest produced by `dbt run` vs `dbt parse` for the same model:

```json
// state/manifest.json (produced by dbt run)
"config": {
"settings": {
"allow_nullable_key": 1,
"replicated_deduplication_window": "0" ← injected at runtime, leaked into manifest
}
}

// target/manifest.json (produced by dbt parse / dbt ls)
"config": {
"settings": {
"allow_nullable_key": 1
← key absent
}
}
```

In a project with 12 affected models this caused their entire downstream graph to be re-run on every CI pipeline, triggered by an unrelated change (e.g. editing a documentation `.yml` file).

### Fix

In `dbt/adapters/clickhouse/impl.py`, deep-copy the settings dict before passing it to `update_model_settings` so the in-memory model config is never mutated:

```python
# Before
settings = model['config'].get('settings', {})

# After
settings = copy.deepcopy(model['config'].get('settings', {}))
```

### dbt and/or ClickHouse server logs

Not applicable — the bug is silent. No warnings or errors are emitted; models are simply re-run unnecessarily on every deferred pipeline run.

### Configuration

#### Environment

* dbt version: 1.10.15
* dbt-clickhouse version: 1.10.1
* clickhouse-connect version (if using http): 0.15.1
* Python version: 3.13.13
* Operating system: macOS

#### ClickHouse server

* ClickHouse Server version: 26.2.1.455
* ClickHouse Server non-default settings: none relevant
* Engine of affected tables: `SharedMergeTree` (ClickHouse Cloud)

Contributor guide

Open the contributing guide

Research direction

Start in dbt/adapters/clickhouse/impl.py at get_model_settings and trace how its settings argument reaches update_model_settings. Verify the change with the supplied dbt run, dbt parse, and deferred state-selection reproduction; done means runtime settings no longer appear in manifest.json and an unchanged deferred run selects zero models.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.