ClickHouse / ClickHouse/dbt-clickhouse

`_check_atomic_exchange` probe leaks orphan tables when cleanup fails (CH Cloud, v1.10.0)

Open
#641 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

## Summary

The `_check_atomic_exchange` probe in `dbclient.py` leaves `__dbt_exchange_test_<0|1>_` tables behind on the failure side — when a DROP in the cleanup loop raises `DbtDatabaseError`, the rest of the loop is skipped and the remaining probe table(s) leak.

The probe also runs **once per connection** (so many times per `dbt run`), so any environment that fails the EXCHANGE accumulates orphans linearly with run count. In one Argent/dbt-clickhouse pipeline against ClickHouse Cloud (v25.12.1.1497) on a daily refresh schedule, this produced **108 orphan `__dbt_exchange_test_*` tables in a single database** over ~2 weeks before we noticed and added a periodic cleanup sweep.

## Environment

- `dbt-clickhouse`: 1.10.0
- `dbt-core`: 1.11.7
- ClickHouse Cloud: 25.12.1.1497 (Shared / SharedMergeTree)
- Python: 3.12

## Where

[`dbt/adapters/clickhouse/dbclient.py::_check_atomic_exchange`](https://github.com/ClickHouse/dbt-clickhouse/blob/v1.10.0/dbt/adapters/clickhouse/dbclient.py#L227-L259)

## What happens

The probe creates two `__dbt_exchange_test_<0|1>_` tables, attempts `EXCHANGE TABLES`, and cleans up in a `finally`:

```python
finally:
try:
for table in swap_tables:
self.command(f'DROP TABLE IF EXISTS {table}')
except DbtDatabaseError:
logger.info('Unexpected server exception dropping table')
```

The `try` wraps the whole `for` loop — if the first DROP raises (we see this in CH Cloud; the root cause of the DROP failure itself is unclear, possibly transient connection state after an EXCHANGE rejection), the second DROP never runs. Both probe tables leak.

PR #543 made the probe applicable to `Shared` engines (good), but didn't change the cleanup-loop shape or cache the probe result.

## Repro

Any dbt project against a CH Cloud workspace where `_check_atomic_exchange` raises during cleanup. We consistently observe `__dbt_exchange_test_*` tables persisting in `system.tables` after `dbt run` completes; the count grows by ~2 per affected connection per invocation.

## Suggested fix

Three independent improvements — any one helps; ideally all three:

1. **Per-iteration cleanup**: move the `try/except DbtDatabaseError` inside the `for` loop so a single DROP failure doesn't abort the rest.
2. **Cache the probe result on the adapter instance**: `_check_atomic_exchange` is deterministic per session, so re-running it many times per `dbt run` is wasted work.
3. **Optionally short-circuit on CH Cloud**: `SELECT value FROM system.settings WHERE name = 'cloud_mode'` returns `'1'` on Cloud; on Shared databases the EXCHANGE is known-supported, so the probe can be skipped entirely. (Less safe if CH Cloud ever changes that guarantee; (1)+(2) alone are sufficient to stop the leak.)

I lean toward (1) + (2). Happy to defer to your preference.

## How we'd like to help

If a fix is already in flight, happy to test against our Cloud workspace. Otherwise, glad to send a PR with whichever combination of (1)/(2)/(3) you prefer — please let us know the project's preferred direction.

Related symptom from a different code path: leaks of `__dbt_new_data_` tables from the `delete+insert` incremental strategy, filed separately.

Thanks for the great adapter.

Contributor guide

Open the contributing guide

Research direction

Start in dbt/adapters/clickhouse/dbclient.py at _check_atomic_exchange and reproduce the cleanup failure described for the probe tables. Implement the preferred per-iteration cleanup and probe-result caching, then verify that one failed DROP does not prevent other tables from being removed and repeated checks do not rerun the probe.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.