ClickHouse / ClickHouse/dbt-clickhouse
Insert query that is OOM'ing in Clickhouse is returning OK to dbt
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
I have an insert query that is consisently out of memory-ing. In the Clickhouse logs I can see an exception 241 being thrown on the query.
In dbt, I see:
```
13:36:37 dbt_clickhouse adapter: SQL status: OK in 136.20 seconds
```
In query_log I see:
```
Code: 241. DB::Exception: Memory limit (total) exceeded: would use 146.96 GiB (attempt to allocate chunk of 68720781687 bytes), maximum: 100.80 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: While executing FillingRightJoinSide. (MEMORY_LIMIT_EXCEEDED) (version 24.5.1.22957 (official build))
```
and the problem is that dbt then proceeds despite reaching the OOM -- I would expect dbt to fail here. This seems to have started to happen on/around September 4 or so.
Contributor guide
Research direction
Start by tracing the Python adapter path that executes insert queries and maps ClickHouse exceptions to dbt's reported SQL status. Reproduce the issue with an insert that exceeds ClickHouse's memory limit and verify that dbt reports failure rather than OK when exception 241 occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100