airbytehq / airbytehq/airbyte

[source-clickhouse] MEMORY_LIMIT_EXCEEDED on large S3-backed table due to unbatched full table scan

未關閉
#76,178 3 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/connectors autoteam community connectors/source/clickhouse needs-triage team/extensibility type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Connector Name

source-clickhouse (ClickHouse v0.3.0)

### Connector Version

v0.3.0

### What step the error happened?

During the sync

### Relevant information

We are syncing a large ClickHouse table (`default.usage_events`) using the ClickHouse source connector. The sync consistently fails with MEMORY_LIMIT_EXCEEDED before completing.

**Environment:**
- ClickHouse version: 25.1.2.1.1426
- Table: default.usage_events
- Disk type: s3WithKeeperDisk (S3-backed storage)
- Connector version: (same as above)
- Sync mode: Incremental

**Error:**
Code 241: MEMORY_LIMIT_EXCEEDED — query killed by OvercommitTracker after reaching the 21.60 GiB memory limit while reading column `dimensions` from table `default.usage_events` (part: all_0_349248029_100052_382803101, starting at mark 59080).

**Root cause:**
The connector issues a full unbatched table scan with no chunking by time range, cursor value range, or row count. For large S3-backed tables, this exhausts instance memory before the query completes. The `dimensions` column (avg ~29.7 chars/row) is the primary memory driver.

**What we've tried:**
- Attempted to pass `max_memory_usage` and related settings via `jdbc_url_params` — confirmed unreliable with the current JDBC driver (per Airbyte support, PR #75432 is in progress to fix this)
- Confirmed with Airbyte support that the connector does not support batched/chunked reads

**Feature requests / expected behavior:**
1. Support chunked reads — split queries by cursor value range or row count (similar to how other JDBC-based connectors handle large tables)
2. Reliable passthrough of ClickHouse server settings (`max_memory_usage`, `max_rows_to_read`, `max_bytes_to_read`) via `jdbc_url_params` once PR #75432 lands

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11927

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。