[source-clickhouse] MEMORY_LIMIT_EXCEEDED on large S3-backed table due to unbatched full table scan
- Lenguaje dominante
- Python
- Estrellas
- 22.1k
- Forks
- 5.3k
- Métricas de merge de PR
- Métricas de PR pendientes
Descripción
### Connector Name
source-clickhouse (ClickHouse v0.3.0)
### Connector Version
v0.3.0
### What step the error happened?
During the sync
### Relevant information
We are syncing a large ClickHouse table (`default.usage_events`) using the ClickHouse source connector. The sync consistently fails with MEMORY_LIMIT_EXCEEDED before completing.
**Environment:**
- ClickHouse version: 25.1.2.1.1426
- Table: default.usage_events
- Disk type: s3WithKeeperDisk (S3-backed storage)
- Connector version: (same as above)
- Sync mode: Incremental
**Error:**
Code 241: MEMORY_LIMIT_EXCEEDED — query killed by OvercommitTracker after reaching the 21.60 GiB memory limit while reading column `dimensions` from table `default.usage_events` (part: all_0_349248029_100052_382803101, starting at mark 59080).
**Root cause:**
The connector issues a full unbatched table scan with no chunking by time range, cursor value range, or row count. For large S3-backed tables, this exhausts instance memory before the query completes. The `dimensions` column (avg ~29.7 chars/row) is the primary memory driver.
**What we've tried:**
- Attempted to pass `max_memory_usage` and related settings via `jdbc_url_params` — confirmed unreliable with the current JDBC driver (per Airbyte support, PR #75432 is in progress to fix this)
- Confirmed with Airbyte support that the connector does not support batched/chunked reads
**Feature requests / expected behavior:**
1. Support chunked reads — split queries by cursor value range or row count (similar to how other JDBC-based connectors handle large tables)
2. Reliable passthrough of ClickHouse server settings (`max_memory_usage`, `max_rows_to_read`, `max_bytes_to_read`) via `jdbc_url_params` once PR #75432 lands
### Relevant log output
```shell
```
### Contribute
- [ ] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11927
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.