ClickHouse / ClickHouse/dbt-clickhouse

`delete+insert` incremental leaks `<model>__dbt_new_data_<uuid>` on failure (v1.10.0)

Open
#642 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

## Summary

The `delete+insert` incremental strategy in the `incremental` materialization creates a `__dbt_new_data_` temp table, but the cleanup `adapter.drop_relation(new_data_relation)` is only on the success path. Any failure between create and drop — query timeout, OOM, manual cancel, network blip — leaks the temp permanently.

Because the temp's name embeds `invocation_id`, the next run can't clean it up via `drop_relation_if_exists` (the name is different on every invocation). Once leaked, the temp is stuck until something pattern-matches and drops it.

In one Argent/dbt-clickhouse pipeline against ClickHouse Cloud (v25.12.1.1497) running on a daily refresh schedule, this produced **130 orphan `__dbt_new_data_*` tables in a single database** over ~2 weeks before we noticed and added a periodic cleanup sweep.

## Environment

- `dbt-clickhouse`: 1.10.0
- `dbt-core`: 1.11.7
- ClickHouse Cloud: 25.12.1.1497 (Shared / SharedMergeTree)
- Python: 3.12

## Where

[`dbt/include/clickhouse/macros/materializations/incremental/incremental.sql`](https://github.com/ClickHouse/dbt-clickhouse/blob/v1.10.0/dbt/include/clickhouse/macros/materializations/incremental/incremental.sql) — the `delete+insert` branch around lines 204–246.

## What happens

```jinja
{% set new_data_relation = existing_relation.incorporate(path={"identifier":
existing_relation.identifier + '__dbt_new_data_' + invocation_id.replace('-', '_')}) %}
{{ drop_relation_if_exists(new_data_relation) }}
...
{% call statement('create_new_data_temp') %}
{{ get_create_table_as_sql(False, new_data_relation, sql) }}
...
{% do adapter.drop_relation(new_data_relation) %}
```

There's no `try/finally` around the create→insert→drop sequence. A failure mid-sequence leaves `__dbt_new_data_` orphaned.

A second variant exists at L127 (`__dbt_new_data` with no invocation_id suffix). That one is self-healing — the next run does `drop_relation_if_exists` on the deterministic name. But the L204 invocation-id-suffixed variant is the one that accumulates.

## Repro

Any model with `incremental_strategy='delete+insert'`. Run, then `kill` the dbt process (or trigger any failure) between the `create_new_data_temp` and `insert_new_data` calls. The `__dbt_new_data_` will remain in `system.tables` indefinitely.

## Suggested fix

Wrap the create→insert sequence in a Jinja-side `try/finally` that always calls `drop_relation_if_exists(new_data_relation)`. Same fix for the `distributed_new_data_relation` cleanup.

Alternatively, swap the manual sequence for an `EXCHANGE TABLES` (atomic) — but that's a larger change and not all engines support EXCHANGE on Cloud.

## How we'd like to help

If a fix is already in flight, happy to test against our Cloud workspace. Otherwise, glad to scope a PR if you have a preferred direction — the materialization layer has more nuance than the probe (distributed variant, other strategies that may share the same shape) so we'd appreciate a maintainer pointer on scope before opening one.

Related symptom from a different code path: probe-table leak in `_check_atomic_exchange` — see #641.

Thanks for the great adapter.

Contributor guide

Open the contributing guide

Research direction

Start in dbt/include/clickhouse/macros/materializations/incremental/incremental.sql, especially the delete+insert branch around lines 204–246, and compare it with the deterministic cleanup near line 127. Reproduce the failure between create_new_data_temp and insert_new_data, then verify that both the invocation-suffixed and distributed temporary relations are cleaned up after failures without changing the self-healing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.