ClickHouse / ClickHouse/dbt-clickhouse
`delete+insert` incremental leaks `<model>__dbt_new_data_<uuid>` on failure (v1.10.0)
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
## Summary
The `delete+insert` incremental strategy in the `incremental` materialization creates a `__dbt_new_data_` temp table, but the cleanup `adapter.drop_relation(new_data_relation)` is only on the success path. Any failure between create and drop — query timeout, OOM, manual cancel, network blip — leaks the temp permanently.
Because the temp's name embeds `invocation_id`, the next run can't clean it up via `drop_relation_if_exists` (the name is different on every invocation). Once leaked, the temp is stuck until something pattern-matches and drops it.
In one Argent/dbt-clickhouse pipeline against ClickHouse Cloud (v25.12.1.1497) running on a daily refresh schedule, this produced **130 orphan `__dbt_new_data_*` tables in a single database** over ~2 weeks before we noticed and added a periodic cleanup sweep.
## Environment
- `dbt-clickhouse`: 1.10.0
- `dbt-core`: 1.11.7
- ClickHouse Cloud: 25.12.1.1497 (Shared / SharedMergeTree)
- Python: 3.12
## Where
[`dbt/include/clickhouse/macros/materializations/incremental/incremental.sql`](https://github.com/ClickHouse/dbt-clickhouse/blob/v1.10.0/dbt/include/clickhouse/macros/materializations/incremental/incremental.sql) — the `delete+insert` branch around lines 204–246.
## What happens
```jinja
{% set new_data_relation = existing_relation.incorporate(path={"identifier":
existing_relation.identifier + '__dbt_new_data_' + invocation_id.replace('-', '_')}) %}
{{ drop_relation_if_exists(new_data_relation) }}
...
{% call statement('create_new_data_temp') %}
{{ get_create_table_as_sql(False, new_data_relation, sql) }}
...
{% do adapter.drop_relation(new_data_relation) %}
```
There's no `try/finally` around the create→insert→drop sequence. A failure mid-sequence leaves `__dbt_new_data_` orphaned.
A second variant exists at L127 (`__dbt_new_data` with no invocation_id suffix). That one is self-healing — the next run does `drop_relation_if_exists` on the deterministic name. But the L204 invocation-id-suffixed variant is the one that accumulates.
## Repro
Any model with `incremental_strategy='delete+insert'`. Run, then `kill` the dbt process (or trigger any failure) between the `create_new_data_temp` and `insert_new_data` calls. The `__dbt_new_data_` will remain in `system.tables` indefinitely.
## Suggested fix
Wrap the create→insert sequence in a Jinja-side `try/finally` that always calls `drop_relation_if_exists(new_data_relation)`. Same fix for the `distributed_new_data_relation` cleanup.
Alternatively, swap the manual sequence for an `EXCHANGE TABLES` (atomic) — but that's a larger change and not all engines support EXCHANGE on Cloud.
## How we'd like to help
If a fix is already in flight, happy to test against our Cloud workspace. Otherwise, glad to scope a PR if you have a preferred direction — the materialization layer has more nuance than the probe (distributed variant, other strategies that may share the same shape) so we'd appreciate a maintainer pointer on scope before opening one.
Related symptom from a different code path: probe-table leak in `_check_atomic_exchange` — see #641.
Thanks for the great adapter.
Contributor guide
Research direction
Start in dbt/include/clickhouse/macros/materializations/incremental/incremental.sql, especially the delete+insert branch around lines 204–246, and compare it with the deterministic cleanup near line 127. Reproduce the failure between create_new_data_temp and insert_new_data, then verify that both the invocation-suffixed and distributed temporary relations are cleaned up after failures without changing the self-healing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100