ClickHouse / ClickHouse/dbt-clickhouse
`replicated_deduplication_window=0` is injected into manifest, breaking deferred runs
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
`get_model_settings` passes the `settings` dict from `model['config']` directly into `update_model_settings`, which mutates it in-place by injecting `replicated_deduplication_window='0'`. Because dbt serialises `manifest.json` after `dbt run`, this injected key gets persisted into the manifest.
The consequence is that commands like `dbt parse` or `dbt ls` — which never call `get_model_settings` — produce a clean manifest without the injected key. When using `--select state:modified+ --defer --state `, dbt compares the two manifests and sees `config_changed` on every affected model, even when nothing actually changed.
Related to #372 — that issue covers the unwanted DDL injection; this reports the deeper consequence of the in-place mutation leaking into the manifest.
### Steps to reproduce
```python
{{ config(
materialized='table',
engine='MergeTree()',
order_by='id',
settings={'allow_nullable_key': 1}
) }}
select 1 as id
```
1. `dbt run --select my_model` — produces `target/manifest.json` with `replicated_deduplication_window='0'` inside `config.settings`
2. `mkdir state && cp target/manifest.json state/`
3. `dbt run --select state:modified+ --defer --state state/` with no code changes
### Expected behaviour
Step 3 selects zero models — nothing changed. Actual: `my_model` is selected and re-run because dbt sees `config_changed` (the injected key is present in the state manifest but absent from the freshly parsed one).
### Code examples, such as models or profile settings
The diff between the manifest produced by `dbt run` vs `dbt parse` for the same model:
```json
// state/manifest.json (produced by dbt run)
"config": {
"settings": {
"allow_nullable_key": 1,
"replicated_deduplication_window": "0" ← injected at runtime, leaked into manifest
}
}
// target/manifest.json (produced by dbt parse / dbt ls)
"config": {
"settings": {
"allow_nullable_key": 1
← key absent
}
}
```
In a project with 12 affected models this caused their entire downstream graph to be re-run on every CI pipeline, triggered by an unrelated change (e.g. editing a documentation `.yml` file).
### Fix
In `dbt/adapters/clickhouse/impl.py`, deep-copy the settings dict before passing it to `update_model_settings` so the in-memory model config is never mutated:
```python
# Before
settings = model['config'].get('settings', {})
# After
settings = copy.deepcopy(model['config'].get('settings', {}))
```
### dbt and/or ClickHouse server logs
Not applicable — the bug is silent. No warnings or errors are emitted; models are simply re-run unnecessarily on every deferred pipeline run.
### Configuration
#### Environment
* dbt version: 1.10.15
* dbt-clickhouse version: 1.10.1
* clickhouse-connect version (if using http): 0.15.1
* Python version: 3.13.13
* Operating system: macOS
#### ClickHouse server
* ClickHouse Server version: 26.2.1.455
* ClickHouse Server non-default settings: none relevant
* Engine of affected tables: `SharedMergeTree` (ClickHouse Cloud)
Contributor guide
Research direction
Start in dbt/adapters/clickhouse/impl.py at get_model_settings and trace how its settings argument reaches update_model_settings. Verify the change with the supplied dbt run, dbt parse, and deferred state-selection reproduction; done means runtime settings no longer appear in manifest.json and an unchanged deferred run selects zero models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100